Low-light video enhancement method based on spatio-temporal feature alignment and illumination space modeling

Abstract

The application discloses a weak light video enhancement method based on spatio-temporal feature alignment and illumination space modeling, and specifically comprises the following steps: step 1, dividing a weak light video into a plurality of continuous weak light video frame sequences; step 2, constructing a network model based on spatio-temporal feature alignment and illumination space modeling, including an input module, a spatio-temporal feature progressive alignment module, an illumination feature extraction module and an illumination estimation network connected in sequence, wherein the output of the spatio-temporal feature progressive alignment module is further connected with the input of a noise estimation network, and the outputs of the illumination estimation network and the noise estimation network are further commonly connected with an enhanced video output module; step 3, training the network model constructed in step 2 by using the weak light video frame sequences; and step 4, performing weak light video enhancement by using the trained network model. The application solves the problems of white-out at strong light and detail loss at dark light caused by uneven illumination distribution in the weak light video enhancement process in the prior art, and the problems of artifacts and color distortion.

Description

Technical Field

[0001] This invention belongs to the technical field of computer digital image processing methods, and relates to a low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling. Background Technology

[0002] With the development of autonomous driving, smart city surveillance, and mobile photography technologies, the imaging quality of camera equipment in complex lighting environments is becoming increasingly important. However, limited by sensor performance, videos captured in extremely low light or strong backlight environments often suffer from low visibility and insufficient contrast. Low-light video enhancement not only requires maintaining stability in the spatiotemporal dimensions and strictly maintaining temporal consistency between frames to avoid flickering, but the core challenge lies in how to reasonably restore and enhance illumination across multiple consecutive frames.

[0003] Although existing technologies attempt to restore illumination through feature mapping, the lack of global explicit modeling of spatial illumination distribution easily leads to whitening in bright areas and loss of detail in dim areas. Furthermore, the lack of a decoupling mechanism between brightness and color may cause artifacts and color distortion while significantly improving video frame quality. Summary of the Invention

[0004] The purpose of this invention is to provide a low-light video enhancement method based on spatiotemporal feature alignment and illumination space modeling, which solves the problems of whitening in strong light areas and loss of details in dark light areas caused by uneven illumination distribution in the low-light video enhancement process of existing technologies, as well as the problems of artifacts and color distortion.

[0005] The technical solution adopted in this invention is a low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling, which is implemented according to the following steps: Step 1: Divide the low-light video into several consecutive low-light video frame sequences; Step 2: Construct a network model based on spatiotemporal feature alignment and illumination spatial modeling, including an input module, a spatiotemporal feature progressive alignment module, an illumination feature extraction module, and an illumination estimation network connected in sequence. The output of the spatiotemporal feature progressive alignment module is also connected to the input of the noise estimation network. The outputs of the illumination estimation network and the noise estimation network are also connected to the video enhancement output module. Step 3: Train the network model constructed in Step 2 using a sequence of low-light video frames; Step 4: Use the trained network model to enhance low-light videos.

[0006] Preferably, step 1 specifically includes: The low-light video is divided into several consecutive low-light video frame sequences. Each low-light video frame sequence consists of five consecutive low-light images, and each low-light video frame sequence is represented as: { },in, This represents a low-light image in a low-light video frame sequence.

[0007] Preferably, the input to the input module in step 2 is a sequence of low-light video frames, and the processing of the input low-light video frame sequence by the input module is as follows: Extract content features from the input low-light video frame sequence, including forward features. Backward features Central features and motion characteristics, including short-range motion characteristics. Long-range dynamic characteristics Structural motion characteristics .

[0008] Preferably, the spatiotemporal feature progressive alignment module takes content features and motion features as input, and performs three operations at three scales on the content features and motion features, denoted as the (s-1), s, and s+1th operations respectively. For each operation, the spatiotemporal feature progressive alignment module outputs the corresponding state features. For the s-th operation, specifically: Step 2.1.1: Perform downsampling at the scale corresponding to the s-th operation on the input content features and motion features to obtain the corresponding downsampled features: , , , and This represents the forward feature, backward feature, and center feature corresponding to the scale downsampled to the s-th time. , and This represents the short-range motion features, long-range dynamic features, and structural motion features that have been downsampled to the scale corresponding to the sth time. Step 2.1.2: For the downsampling result of the s-th execution, the downsampling features of the content features are concatenated along the channel dimension. The spatiotemporal information is initially fused through a convolutional layer and the LeakyReLU activation function. The state features output by the spatiotemporal feature progressive alignment module from the previous execution are introduced through residual connections. , to obtain features Specifically:

[0009] Where [] represents feature concatenation, Conv() represents convolution operation; LeakyReLU() represents LeakyReLU activation function, and ⊕ represents residual connection; During the (s-1)th execution, That is, it only requires downsampling the forward features obtained at the scale corresponding to the (s-1)th operation. Backward features Central features By concatenating the data along the channel dimension and then using a convolutional layer and the LeakyReLU activation function to achieve preliminary fusion of spatiotemporal information, the corresponding features can be obtained. ; Step 2.1.3: The motion features corresponding to the downsampling performed for the s-th time are mapped to the Sigmoid activation function through a convolutional layer to generate a motion guidance mask. Specifically:

[0010] Where σ() represents the Sigmoid activation function; Step 2.1.4, using motion-guided masks Features Modulation is performed to obtain the offset. :

[0011] in, ReLU represents pointwise multiplication, and ReLU represents the ReLU activation function. Through the Channel segmentation yields two offsets, denoted as follows: ; according to Forward features in content features Backward features Perform a transformation operation to transform the forward features With backward features Perform transformations to obtain aligned features :

[0012] Where Warp represents the transformation operation; Step 2.1.5, align the features Features of the original center The final aggregation is performed to obtain the aggregated features. :

[0013] Where GELU() represents the GELU activation function; Step 2.1.6, aggregate features The state features output by the spatiotemporal feature progressive alignment module at the s-th execution are adjusted through convolutional layers. : ; Step 2.1.7: Perform the steps 2.1.1-2.1.6 sequentially at the three scales a total of three times, and output the result of the (s+1)th time. The final output of the spatiotemporal feature progressive alignment module is obtained by adjusting the convolutional layers: spatiotemporal alignment features. .

[0014] Preferred forward features Backward features Central features Specifically, it is obtained in the following manner: For the first two frames in a low-light video frame sequence , The low-light images are processed by extracting corresponding features through two layers of 3×3 residual convolutional blocks with shared weights. , Then , The features are then concatenated and fused using a 1×1 convolutional layer to obtain the final result. ,according to Obtain forward features containing local motion trends :

[0015] Where Conv represents a convolutional layer, GELU represents the GELU activation function, and [] represents the feature concatenation operation; For the last two frames in a low-light video frame sequence , For each frame of low-light image, the corresponding features are extracted through two 3×3 residual convolutional blocks with shared weights. , Then , The features are then concatenated and fused using a 1×1 convolutional layer to obtain the final result. ,according to Obtain backward features :

[0016] For the center frame The central features are extracted by passing them through three cascaded residual convolutional blocks. :

[0017] ResBlocks represents three layers of cascaded residual blocks.

[0018] Preferred short-range motion characteristics Long-range dynamic characteristics Structural motion characteristics Specifically, it is obtained in the following manner: Short-range motion features are calculated by the difference between adjacent frames:

[0019] Where || represents absolute value; Long-range dynamic features are calculated using cross-frame differencing:

[0020] Extracting structural motion features using the Sobel operator:

[0021] Sobel() represents the Sobel operator function.

[0022] Preferably, in step 2, the illumination feature extraction module uses spatiotemporally aligned features. As input, the specific working process is as follows: Step 2.2.1, process the input spatiotemporal alignment features Features are generated sequentially through global average pooling, sigmoid activation, and channel attention. Specifically:

[0023] Where GAP() represents global average pooling operation, CA() represents channel attention operation, and σ() represents Sigmoid activation function processing; Step 2.2.2: Use convolutional layers to process the features. Channel segmentation is performed, resulting in two features. , of which F lum For brightness features, used to process brightness and geometry; F chorm These are color characteristics used to maintain color consistency; Step 2.2.3, feature Input is fed into a window-based multi-head self-attention branch, configured for different channels. Construct multi-scale spatial masks to obtain a unified spatial attention representation A. spatial :

[0024] Where H and W represent F lum Height and width, Indicates the brightness feature F lum Execute multiple parallel convolutional layers with different numbers of channels, where the number of channels is determined by n. f Sure; Step 2.2.4, represent spatial attention as Aspatial Then, in the window-based multi-head self-attention branch, the attention weight matrix is ​​obtained based on the query matrix Q, the key matrix K, and the value matrix V. And illumination-sensing feature vectors:

[0025]

[0026] Where Q and K are determined by brightness characteristics Normalization is achieved by passing the normalized result through parallel convolutional layers; d represents the scaling factor; B is the learnable relative position bias; Softmax represents the normalization function; V is derived from color features. It is obtained through normalization and convolutional layers; Step 2.2.5, using illumination-sensing feature vectors and brightness features Obtain features :

[0027] in, This represents the random depth dropout operator, where γ is the scaling factor for the convolution branch; Step 2.2.6, will First, normalization is performed, followed by upsampling and downsampling, with a GELU activation function introduced between the upsampling and downsampling. Then, random discarding is performed to finally generate depth illumination-aware features. :

[0028] Where LN represents layer normalization, L1 and L2 represent upsampling and downsampling respectively; DP() represents the random drop function; Step 2.2.7, perceive depth illumination features ,feature Spatiotemporal alignment features Illumination characteristics are obtained by performing residual connection. :

[0029] Where λ is set to 0.1.

[0030] Preferably, the illumination estimation network uses illumination features As input, illumination features After passing through N residual connection layers, followed by upsampling layers, convolutional layers, and activation functions, the final illumination map is output. ; Noise estimation network with spatiotemporal alignment features As input, spatiotemporal alignment features The estimated noise is obtained after passing through N residual connection layers, upsampling layers, and convolutional layers in sequence. .

[0031] Preferably, the enhanced video output module uses a lighting map. and noise estimation As input, a normal lighting frame is obtained using the following formula. :

[0032] Among them, I t The center frame of the input low-light video frame sequence; The normal lighting frames corresponding to each low-light video frame sequence are arranged in order to form a complete enhanced video output.

[0033] Preferably, during training in step 3, a sequence of low-light video frames, including five consecutive low-light images, is used as input. The batch size is set to 32, the input cropping size is 128×128, and the Adam optimizer is used. The momentum parameters β1 and β2 are set to 0.9, and the initial learning rate is set to 4×10⁻⁶. -4 The cosine annealing learning rate restart scheduling strategy was adopted for optimization. The total number of training iterations was set to 150,000, and the learning rate was restarted and decayed according to the set period during the training process. Step 4 specifically involves dividing the low-light video to be enhanced into several low-light video frame sequences, each consisting of five consecutive low-light images. These low-light video frame sequences are then sequentially input into the network model trained in Step 3 to obtain the complete enhanced video.

[0034] The beneficial effects of this invention are: To address the challenges of dynamic scene alignment in low-light environments, the extremely low energy of light components leading to whitening in bright areas and loss of detail in dark areas, accompanied by severe random noise and color distortion, this invention constructs a collaborative mechanism of progressive spatiotemporal feature alignment and illumination space modeling. First, a progressive spatiotemporal feature alignment module is built to progressively align dynamic targets using temporal information, suppressing ghosting and flickering artifacts. Based on this, the aligned spatiotemporal features are decomposed into brightness and color features, and by applying multi-scale spatial mask weighting to the spatial illumination distribution, explicit capture of complex illumination distributions and color fidelity in the feature flow is achieved. This collaborative optimization strategy, starting from the dual dimensions of progressive spatiotemporal alignment and decoupled color space modeling, effectively solves the problems of whitening in bright areas and loss of detail in dark areas caused by uneven illumination distribution in existing low-light video enhancement techniques, while also addressing the artifacts and severe color cast issues common in traditional methods. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the network model based on spatiotemporal feature alignment and illumination space modeling in the low-light video enhancement method based on spatiotemporal feature alignment and illumination space modeling of the present invention; Figure 2 This is a schematic diagram of the spatiotemporal feature progressive alignment module in the low-light video enhancement method based on spatiotemporal feature alignment and illumination space modeling of the present invention; Figure 3 This is a comparison diagram between the method of the present invention and existing methods; Figure 4 This is a comparison diagram of the effects of the present invention and the method using event camera information; Figure 5 This is a visualization comparison of the method of the present invention and existing low-light enhancement methods on the SMID dataset in Example 6. Detailed Implementation

[0036] The following detailed description is provided in conjunction with specific implementation methods.

[0037] Example 1 This invention relates to a low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling, which is implemented according to the following steps: Step 1: Divide the low-light video into several consecutive low-light video frame sequences; Step 2: Construct a network model based on spatiotemporal feature alignment and illumination space modeling, the structure of which is as follows: Figure 1 As shown, it includes an input module, a spatiotemporal feature progressive alignment module, an illumination feature extraction module, and an illumination estimation network connected in sequence. The output of the spatiotemporal feature progressive alignment module is also connected to the input of the noise estimation network. The outputs of the illumination estimation network and the noise estimation network are also connected to the enhanced video output module. Step 3: Train the network model constructed in Step 2 using a sequence of low-light video frames; Step 4: Use the trained network model to enhance low-light videos.

[0038] Example 2 Based on Example 1, step 1 specifically includes: The low-light video is divided into several consecutive low-light video frame sequences. Each low-light video frame sequence consists of five consecutive low-light images, and each low-light video frame sequence is represented as: { },in, This represents a low-light image in a low-light video frame sequence.

[0039] Example 3 Based on Example 2, the input to the input module in step 2 is a sequence of low-light video frames. The specific processing of the input low-light video frame sequence by the input module is as follows: Extract content features from the input low-light video frame sequence, including forward features. Backward features Central features and motion characteristics, including short-range motion characteristics. Long-range dynamic characteristics Structural motion characteristics : Forward features Backward features Central features Specifically, it is obtained in the following manner: For the first two frames in a low-light video frame sequence , The low-light images are processed by extracting corresponding features through two layers of 3×3 residual convolutional blocks with shared weights. , Then , The features are then concatenated and fused using a 1×1 convolutional layer to obtain the final result. ,according to Obtain forward features containing local motion trends :

[0040] Where Conv represents a convolutional layer, GELU represents the GELU activation function, and [] represents the feature concatenation operation; For the last two frames in a low-light video frame sequence , For each frame of low-light image, the corresponding features are extracted through two 3×3 residual convolutional blocks with shared weights. , Then , The features are then concatenated and fused using a 1×1 convolutional layer to obtain the final result. ,according to Obtain backward features :

[0041] For the center frame The central features are extracted by passing them through three cascaded residual convolutional blocks. :

[0042] ResBlocks represents three layers of cascaded residual blocks.

[0043] Short-range motion characteristics Long-range dynamic characteristics Structural motion characteristics Specifically, it is obtained in the following manner: Motion information is constructed by calculating inter-frame differences, and short-range motion features are calculated by calculating the differences between adjacent frames.

[0044] Where || represents absolute value; Long-range dynamic features are calculated using cross-frame differencing:

[0045] Extracting structural motion features using the Sobel operator:

[0046] Where Sobel() represents the Sobel operator function; The spatiotemporal feature progressive alignment module takes content features and motion features as input, and performs three operations at three scales on the content features and motion features, denoted as the (s-1), s, and s+1th operations respectively. For each operation, the spatiotemporal feature progressive alignment module outputs the corresponding state features. For the s-th operation, as shown... Figure 2 As shown, specifically: Step 2.1.1: Perform downsampling at the scale corresponding to the s-th operation on the input content features and motion features to obtain the corresponding downsampled features: , , , and This represents the forward feature, backward feature, and center feature corresponding to the scale downsampled to the s-th time. , and This represents the short-range motion features, long-range dynamic features, and structural motion features that have been downsampled to the scale corresponding to the sth time. Step 2.1.2: For the downsampling result of the s-th execution, the downsampling features of the content features are concatenated along the channel dimension. The spatiotemporal information is initially fused through a convolutional layer and the LeakyReLU activation function. The state features output by the spatiotemporal feature progressive alignment module from the previous execution are introduced through residual connections. , to obtain features Specifically:

[0047] Where [] represents feature concatenation, Conv() represents convolution operation; LeakyReLU() represents LeakyReLU activation function, and ⊕ represents residual connection, which is used to connect features... The provided prior alignment information is incorporated into the features of the current stage s; During the (s-1)th execution, That is, it only requires downsampling the forward features obtained at the scale corresponding to the (s-1)th operation. Backward features Central features By concatenating the data along the channel dimension and then using a convolutional layer and the LeakyReLU activation function to achieve preliminary fusion of spatiotemporal information, the corresponding features can be obtained. ; Step 2.1.3: Simultaneously, spatial attention weights are constructed using motion features to enhance the model's ability to perceive dynamic regions. Based on the motion features corresponding to the downsampling of the s-th execution, a motion guidance mask is generated by mapping the convolutional layer with the Sigmoid activation function. Specifically:

[0048] Where σ() represents the Sigmoid activation function; Step 2.1.4, using motion-guided masks Features Modulation is performed to obtain the offset. :

[0049] in, ReLU represents pointwise multiplication, and ReLU represents the ReLU activation function. Through the Channel segmentation yields two offsets, denoted as follows: ; according to Forward features in content features Backward features Perform a transformation operation to transform the forward features With backward features Perform transformations to obtain aligned features :

[0050] Where Warp represents the transformation operation; Step 2.1.5, align the features Features of the original center The final aggregation is performed to obtain the aggregated features. :

[0051] Where GELU() represents the GELU activation function; Step 2.1.6, aggregate features The state features output by the spatiotemporal feature progressive alignment module at the s-th execution are adjusted through convolutional layers. : ; Step 2.1.7: Perform the steps 2.1.1-2.1.6 sequentially at the three scales a total of three times, and output the result of the (s+1)th time. The final output of the spatiotemporal feature progressive alignment module is obtained by adjusting the convolutional layers: spatiotemporal alignment features. .

[0052] Preferably, in step 2, the illumination feature extraction module uses spatiotemporally aligned features. As input, the specific working process is as follows: Step 2.2.1, process the input spatiotemporal alignment features Features are generated sequentially through global average pooling, sigmoid activation, and channel attention. Specifically:

[0053] Where GAP() represents global average pooling operation, CA() represents channel attention operation, and σ() represents Sigmoid activation function processing; Step 2.2.2: Use convolutional layers to process the features. Channel segmentation is performed, resulting in two features. , of which F lum For brightness features, used to process brightness and geometry; F chorm These are color characteristics used to maintain color consistency; Step 2.2.3, feature Input is fed into a window-based multi-head self-attention branch (W-MSA), configured for different channels. Construct multi-scale spatial masks to obtain a unified spatial attention representation A. spatial :

[0054] Where H and W represent F lum Height and width, Indicates the brightness feature F lum Execute multiple parallel convolutional layers with different numbers of channels, where the number of channels is determined by n. f Sure; Step 2.2.4, represent spatial attention as A spatialThen, in the window-based multi-head self-attention branch, the attention weight matrix is ​​obtained based on the query matrix Q, the key matrix K, and the value matrix V. And illumination-sensing feature vectors:

[0055]

[0056] Where Q and K are determined by brightness characteristics Normalization is achieved by passing the normalized result through parallel convolutional layers; d represents the scaling factor, with a value of 4; B is the learnable relative position bias, initialized to 0; Softmax represents the normalization function; V is derived from color features. It is obtained through normalization and convolutional layers; Step 2.2.5, using illumination-sensing feature vectors and brightness features Obtain features :

[0057] in, This represents the random depth dropout operator, used to improve the model's generalization ability; γ is the convolution branch scaling factor, set to 0.01; Step 2.2.6, will First, normalization is performed, followed by upsampling and downsampling, with a GELU activation function introduced between the upsampling and downsampling. Then, random discarding is performed to finally generate depth illumination-aware features. :

[0058] Where LN represents layer normalization, L1 and L2 represent upsampling and downsampling respectively; DP() represents the random drop function; Step 2.2.7, perceive depth illumination features ,feature Spatiotemporal alignment features Illumination characteristics are obtained by performing residual connection. :

[0059] Where λ is set to 0.1.

[0060] Preferably, the illumination estimation network uses illumination features As input, illumination features After passing through N residual connection layers, followed by upsampling layers, convolutional layers, and activation functions, the final illumination map is output. ; Noise estimation network with spatiotemporal alignment features As input, spatiotemporal alignment features The estimated noise is obtained after passing through N residual connection layers, upsampling layers, and convolutional layers in sequence. .

[0061] Preferably, the enhanced video output module uses a lighting map. and noise estimation As input, a normal lighting frame is obtained using the following formula. :

[0062] Among them, I t The center frame of the input low-light video frame sequence; The normal lighting frames corresponding to each low-light video frame sequence are arranged in order to form a complete enhanced video output.

[0063] Example 4 Based on Example 3, during training in step 3, an existing indoor low-light video dataset was selected for training. A single GPU was chosen for training, with a GPU memory configuration of 24GB. The input consisted of a sequence of low-light video frames including five consecutive low-light images. The batch size was set to 32, and the input cropping size was 128×128. The Adam optimizer was used, with momentum parameters β1 and β2 set to 0.9 and the initial learning rate set to 4×10⁻⁶. -4 The learning rate scheduling strategy was optimized by using cosine annealing to restart the learning rate. The total number of training iterations was set to 150,000. During the training process, the learning rate was restarted and decayed according to the set period to improve the convergence performance and stability of the model.

[0064] In the imaging process of low-light videos, due to insufficient ambient irradiance and the randomness of sensor quantum noise, the original signals often exhibit extreme dynamic range loss, detail annihilation, and severe uneven brightness distribution. Conventional single-frame enhancement algorithms, lacking effective modeling of temporal coherence, are prone to producing inter-frame brightness jumps and visual flicker artifacts when processing dynamic scenes. Furthermore, without accurate mapping of complex spatial illumination, simple linear stretching often leads to local overexposure or complete darkness, making it difficult to restore a natural and consistent visual effect. To address these bottlenecks, this invention constructs a temporal neighborhood feature evolution mechanism centered on a central reference frame. A spatiotemporal feature progressive alignment module captures and compensates for nonlinear motion vectors, thereby aggregating and aligning redundant information from multiple frames in the feature space, significantly improving the temporal continuity of the video enhancement process. At the brightness reconstruction and optimization level, an illumination feature extraction module is used to extract local structural details through convolutional branches and capture long-range illumination dependencies using self-attention branches. Simultaneously, an adaptive multi-scale spatial mask is introduced to differentiate the illumination distribution in different regions. By deeply fusing spatiotemporal features and nonlinearly mapping the illumination space, this invention enables precise illumination compensation and dynamic range extension of the center frame based on a physical imaging model. Ultimately, through synergistic optimization of temporal stability and spatial illumination distribution, this invention achieves significant brightness gain while effectively preserving the image's edge structure and suppressing artifacts, resulting in high-quality enhanced video with natural and balanced illumination, clear visuals, and stable temporal features.

[0065] like Figure 3 As shown, the processing flow of four different low-light video enhancement methods and their corresponding enhancement results are illustrated, with a comparison between the input image and the ground truth (GT). The results show that existing methods (a video enhancement method for seeing dynamic scenes in darkness (SDSD), a low-light video enhancement network based on 3D-UNet (SMOID), and a low-light image enhancement method based on zero-reference depth curve estimation (ZeroDCE)) are prone to problems such as insufficient detail recovery or uneven brightness during the enhancement process due to their failure to effectively coordinate spatiotemporal information or lack of illumination modeling. In contrast, the method of this invention (Ours), through the collaborative optimization of spatiotemporal alignment and illumination space modeling, exhibits superior visual effects in the enhanced image. Not only is the overall brightness more natural and uniform, but it is also more accurate in terms of texture details and color reproduction, with the overall effect closest to the ground truth (GT).

[0066] Example 5 Based on Example 4, step 4 specifically involves: dividing the low-light video to be enhanced into several low-light video frame sequences, each consisting of five consecutive low-light images; and inputting the low-light video frame sequences sequentially into the network model trained in step 3 to obtain the complete enhanced video.

[0067] Example 6 like Figure 4 As shown, Figure 4 It includes low-light video frames from the SDSD indoor dataset, event information extracted from the SDSD indoor dataset, enhancement results from the event camera model, and enhancement results from the method of this invention. For example... Figure 4 As shown, by comparing the processing results of this invention with existing event camera models, it can be found that the enhancement results of models specifically using event camera datasets exhibit significant blurring in color restoration and produce certain artifacts. In contrast, the enhancement results of the method in this invention accurately overcome the whitening and detail loss caused by uneven illumination distribution, and are superior to models specifically using event camera datasets in both brightness and color restoration.

[0068] The method of this invention was compared with several existing low-light video enhancement methods, including: a video enhancement method for seeing dynamic scenes in darkness (SDSD), a zero-reference low-light enhancement method based on physical quadruple priors (ZRLLE-PQP), a low-light video enhancement method based on static video and cross-frame interaction (LLVE-CFA), a low-light video enhancement method based on synthetic event guidance (LLVE-SEG), a CNN-based low-light image / video enhancement method (MBLLEN), a low-light video enhancement network based on 3D-UNet (SMOID), a binarized low-light video enhancement method (BLLRVE), and a joint denoising and adaptive method guided by zero-sample illumination. We presented several low-light image enhancement methods, including JDAE (Journal of Visual Algebraic Image Enhancement), CompEvent (a method for low-light video enhancement and deblurring based on complex-valued events and RGB fusion), and DeepLPF (a method for low-light image enhancement based on deep local filters), PairLLE (a method for low-light image enhancement and anti-wearing based on pairwise low-light image learning), RetinexFormer (a method for low-light image enhancement based on retinal theory), MambaIR (a method for image restoration based on a state-space model), and AttenFlow (a method for low-light image enhancement guided by conditional attention using normalized flow). Figure 5As shown, a visual comparison of different methods was performed using the Seeing Motion in Darkness (SMID) dataset. All results were obtained based on the publicly available models or default settings of each method. The comparison results reveal that some methods suffer from significant noise residue, uneven brightness, or blurred details during the enhancement process. For example, the PairLLE method is prone to oversmoothing or loss of detail when restoring dark areas, while the SMOID method introduces artifacts or color biases during enhancement, affecting the overall visual effect and structural consistency. RetinexFormer performs well in overall restoration, but its results have an overall grayish tone and exhibit some color distortion. In contrast, the method of this invention can effectively restore illumination while maintaining spatiotemporal consistency and preserving color consistency with normal light video, resulting in a more natural and clearer visual enhancement that is closer to the real scene, demonstrating superior subjective visual quality and stability.

[0069] The objective evaluations of the method of this invention and existing methods on the indoor dataset (SDSD-indoor) and the SMID dataset are shown in Tables 1 and 2. As can be seen from Tables 1 and 2, the method of this invention outperforms existing methods in both the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics, demonstrating more stable and significant overall performance. The statistical results show that the method of this invention maintains a leading advantage on different datasets and outperforms existing comparative methods overall, further validating the effectiveness and superiority of the method of this invention in low-light video enhancement tasks.

[0070] Table 1

[0071] Table 2

[0072] This invention presents a low-light video enhancement method based on spatiotemporal feature alignment and illumination space modeling. It proposes a spatiotemporal feature progressive alignment module to effectively utilize neighboring frame information, thereby improving temporal consistency and reducing flickering issues caused by independent frame-by-frame enhancement. An illumination space feature extraction module is also constructed, achieving accurate modeling and structure-aware optimization of illumination distribution through multi-scale spatial mask fusion and bi-branch modeling. By synergistically fusing multi-scale spatiotemporal feature alignment and color space illumination modeling, this invention achieves joint optimization of issues such as ghosting, whitening in bright areas, loss of detail in dark areas, and color distortion in low-light videos. This significantly improves image brightness while effectively maintaining temporal consistency between frames, both forward and backward.

Claims

1. A low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling, characterized in that, The specific steps are as follows: Step 1: Divide the low-light video into several consecutive low-light video frame sequences; Step 2: Construct a network model based on spatiotemporal feature alignment and illumination space modeling, including an input module, a spatiotemporal feature progressive alignment module, an illumination feature extraction module, and an illumination estimation network connected in sequence. The output of the spatiotemporal feature progressive alignment module is also connected to the input of the noise estimation network. The outputs of the illumination estimation network and the noise estimation network are also connected to the enhanced video output module. Step 3: Train the network model constructed in Step 2 using a sequence of low-light video frames; Step 4: Use the trained network model to enhance low-light videos.

2. The low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling according to claim 1, characterized in that, Step 1 specifically involves: The low-light video is divided into several consecutive low-light video frame sequences. Each low-light video frame sequence consists of five consecutive low-light images, and each low-light video frame sequence is represented as: { },in, This represents a low-light image in a low-light video frame sequence.

3. The low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling according to claim 2, characterized in that, The input to the input module in step 2 is a sequence of low-light video frames. The specific processing of the input low-light video frame sequence by the input module is as follows: Extract content features from the input low-light video frame sequence, including forward features. Backward features Central features and motion characteristics, including short-range motion characteristics. Long-range dynamic characteristics Structural motion characteristics .

4. The low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling according to claim 3, characterized in that, The spatiotemporal feature progressive alignment module takes content features and motion features as input, and performs three operations at three scales on the content features and motion features, denoted as the (s-1), s, and s+1th operations respectively. For each operation, the spatiotemporal feature progressive alignment module outputs the corresponding state features. Specifically, for the s-th operation: Step 2.1.1: Perform downsampling at the scale corresponding to the s-th operation on the input content features and motion features to obtain the corresponding downsampled features: , , , and This represents the forward feature, backward feature, and center feature corresponding to the scale downsampled to the s-th time. , and This represents the short-range motion features, long-range dynamic features, and structural motion features that have been downsampled to the scale corresponding to the sth time. Step 2.1.2: For the downsampling result of the s-th execution, the downsampling features of the content features are concatenated along the channel dimension. The spatiotemporal information is initially fused through a convolutional layer and the LeakyReLU activation function. The state features output by the spatiotemporal feature progressive alignment module from the previous execution are introduced through residual connections. , to obtain features Specifically: Where [] represents feature concatenation, Conv() represents convolution operation; LeakyReLU() represents LeakyReLU activation function, and ⊕ represents residual connection; During the (s-1)th execution, That is, it only requires downsampling the forward features obtained at the scale corresponding to the (s-1)th operation. Backward features Central features By concatenating the data along the channel dimension and then using a convolutional layer and the LeakyReLU activation function to achieve preliminary fusion of spatiotemporal information, the corresponding features can be obtained. ; Step 2.1.3: The motion features corresponding to the downsampling performed for the s-th time are mapped to the Sigmoid activation function through a convolutional layer to generate a motion guidance mask. Specifically: Where σ() represents the Sigmoid activation function; Step 2.1.4, using motion-guided masks Features Modulation is performed to obtain the offset. : in, ReLU represents pointwise multiplication, and ReLU represents the ReLU activation function. Through the Channel segmentation yields two offsets, denoted as follows: ; according to Forward features in content features Backward features Perform a transformation operation to transform the forward features With backward features Perform transformations to obtain aligned features : Where Warp represents the transformation operation; Step 2.1.5, align the features Features of the original center The final aggregation is performed to obtain the aggregated features. : Where GELU() represents the GELU activation function; Step 2.1.6, aggregate features The state features output by the spatiotemporal feature progressive alignment module at the s-th execution are adjusted through convolutional layers. : ； Step 2.1.7: Perform the steps 2.1.1-2.1.6 sequentially at the three scales a total of three times, and output the result of the (s+1)th time. The final output of the spatiotemporal feature progressive alignment module is obtained by adjusting the convolutional layers: spatiotemporal alignment features. .

5. The low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling according to claim 4, characterized in that, The forward features Backward features Central features Specifically, it is obtained in the following manner: For the first two frames in a low-light video frame sequence , The low-light images are processed by extracting corresponding features through two layers of 3×3 residual convolutional blocks with shared weights. , Then , The features are then concatenated and fused using a 1×1 convolutional layer to obtain the final result. ,according to Obtain forward features containing local motion trends : Where Conv represents a convolutional layer, GELU represents the GELU activation function, and [] represents the feature concatenation operation; For the last two frames in a low-light video frame sequence , For each frame of low-light image, the corresponding features are extracted through two 3×3 residual convolutional blocks with shared weights. , Then , The features are then concatenated and fused using a 1×1 convolutional layer to obtain the final result. ,according to Obtain backward features : For the center frame The central features are extracted by passing them through three cascaded residual convolutional blocks. : ResBlocks represents three layers of cascaded residual blocks.

6. The low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling according to claim 5, characterized in that, The short-range motion characteristics Long-range dynamic characteristics Structural motion characteristics Specifically, it is obtained in the following manner: Short-range motion features are calculated by the difference between adjacent frames: Where || represents absolute value; Long-range dynamic features are calculated using cross-frame differencing: Extracting structural motion features using the Sobel operator: Sobel() represents the Sobel operator function.

7. The low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling according to claim 6, characterized in that, In step 2, the illumination feature extraction module uses spatiotemporal alignment features. As input, the specific working process is as follows: Step 2.2.1, process the input spatiotemporal alignment features Features are generated sequentially through global average pooling, sigmoid activation, and channel attention. Specifically: Where GAP() represents global average pooling operation, CA() represents channel attention operation, and σ() represents Sigmoid activation function processing; Step 2.2.2: Use convolutional layers to process the features. Channel segmentation is performed, resulting in two features. , of which F lum For brightness features, used to process brightness and geometry; F chorm These are color characteristics used to maintain color consistency; Step 2.2.3, feature Input is fed into a window-based multi-head self-attention branch, configured for different channels. Construct multi-scale spatial masks to obtain a unified spatial attention representation A. spatial : Where H and W represent F lum Height and width, This indicates the brightness feature F lum Execute multiple parallel convolutional layers with different numbers of channels, where the number of channels is determined by n. f Sure; Step 2.2.4, represent spatial attention as A spatial Then, in the window-based multi-head self-attention branch, the attention weight matrix is ​​obtained based on the query matrix Q, the key matrix K, and the value matrix V. And illumination-sensing feature vectors: Where Q and K are determined by brightness characteristics Normalization is achieved by passing the normalized result through parallel convolutional layers; d represents the scaling factor; B is the learnable relative position bias; Softmax represents the normalization function; V is derived from color features. It is obtained through normalization and convolutional layers; Step 2.2.5, using illumination-sensing feature vectors and brightness features Obtain features : in, This represents the random depth dropout operator, where γ is the scaling factor for the convolution branch; Step 2.2.6, will First, normalization is performed, followed by upsampling and downsampling, with a GELU activation function introduced between the upsampling and downsampling. Then, random discarding is performed to finally generate depth illumination-aware features. : Where LN represents layer normalization, L1 and L2 represent upsampling and downsampling respectively; DP() represents the random drop function; Step 2.2.7, perceive depth illumination features ,feature Spatiotemporal alignment features Illumination characteristics are obtained by performing residual connection. : Where λ is set to 0.

1.

8. The low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling according to claim 7, characterized in that, The illumination estimation network uses illumination features As input, illumination features After passing through N residual connection layers, followed by upsampling layers, convolutional layers, and activation functions, the final illumination map is output. ; The noise estimation network uses spatiotemporal alignment features As input, spatiotemporal alignment features The estimated noise is obtained after passing through N residual connection layers, upsampling layers, and convolutional layers in sequence. .

9. The low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling according to claim 7, characterized in that, The enhanced video output module uses a lighting pattern. and noise estimation As input, a normal lighting frame is obtained using the following formula. : Among them, I t The center frame of the input low-light video frame sequence; The normal lighting frames corresponding to each low-light video frame sequence are arranged in order to form a complete enhanced video output.

10. The low-light video enhancement method based on spatiotemporal feature alignment and illumination spatial modeling according to claim 8, characterized in that, In step 3, during training, a sequence of low-light video frames, including five consecutive low-light images, is used as input. The batch size is set to 32, the input cropping size is 128×128, and the Adam optimizer is used. The momentum parameters β1 and β2 are set to 0.9, and the initial learning rate is set to 4×10⁻⁶. -4 The cosine annealing learning rate restart scheduling strategy was adopted for optimization. The total number of training iterations was set to 150,000, and the learning rate was restarted and decayed according to the set period during the training process. Step 4 specifically involves dividing the low-light video to be enhanced into several low-light video frame sequences, each consisting of five consecutive low-light images. These low-light video frame sequences are then sequentially input into the network model trained in step 3 to obtain the complete enhanced video.

Images