A low-light video enhancement method and system based on multi-scale spatio-temporal feature fusion

Abstract

This invention relates to a method and system for enhancing low-light videos based on multi-scale spatiotemporal feature fusion, belonging to the field of computer vision technology. During the training phase, the low-light video is preprocessed according to its different data formats, separating different pixels into different channels using color filters. A low-light video enhancement algorithm generates an enhanced video from the input low-light video and compares it with a real normal-light video. The loss function is calculated, and the parameters in the algorithm model are updated through backpropagation. When a preset termination condition is reached, the model structure and the updated model parameters are saved. During the usage phase, a low-light video enhancement model is constructed based on the model structure and parameters saved during the training phase. The low-light video to be processed is then fed into the model for low-light enhancement. This invention can restore low-light videos to high-quality normal-light videos with excellent evaluation metrics and good subjective visual experience.

Description

TECHNICAL FIELD

[0001] The present application relates to a dark light video enhancement method and system, in particular to a dark light video enhancement method and system based on multi-scale spatio-temporal feature fusion, belonging to the technical field of computer vision. BACKGROUND

[0002] Dark light video refers to a video obtained by shooting under suboptimal lighting conditions due to unavoidable environmental or technical limitations (such as insufficient lighting or limited exposure time). Such videos are affected by backlight, uneven light, and dim light, and have drawbacks such as low contrast and high noise, resulting in poor visual experience. In addition, due to darkness and noise hiding details in the image, the amount of information carried by the image is greatly reduced, which has many adverse effects on a series of downstream fields of computer vision, such as target detection and instance segmentation.

[0003] Dark light video enhancement technology aims to improve the perceptual or interpretability of dark light video, and is an important video processing technology. Through dark light enhancement, noise in the video can be removed, brightness can be increased, contrast can be enhanced, and detail levels can be restored to restore dark light video to normal lighting video. With the continuous development of dark light video enhancement technology, some problems that are difficult to solve from the hardware level have made great progress with the support of dark light enhancement algorithms. This technology is widely used in fields such as mobile phone computational photography, autonomous driving, and night surveillance, creating great practical value and economic value, and achieving satisfactory results.

[0004] Traditional dark light video enhancement technology mainly uses prior constraints or statistical principles to establish algorithm models to improve the performance of dark light video. For example, methods such as SSR use the Retinex theory as a prior constraint to decompose dark light images into illumination components and reflection components, and correct standard lighting based on this. The histogram equalization method is to count the distribution of pixel light intensity in the image, and to balance the intensity of the entire image to meet the form of approximately uniform distribution, to realize the brightening of the image or video. The above methods are widely used in the bottom processing flow of image acquisition devices, or as upstream algorithms of major video processing algorithms to preprocess data.

[0005] However, traditional algorithms rely on the effectiveness of model assumptions, and need to manually adjust algorithm models based on different prior conditions for specific tasks, greatly limiting processing capacity. In addition, most traditional dark light enhancement algorithms are limited to improving video brightness and contrast, and are insufficient in suppressing noise, resulting in unsatisfactory visual effects after enhancement, and even more unable to serve as excellent input data for downstream visual algorithms.

[0006] In recent years, with the rapid development of deep learning technology, we have constructed a deep learning network model with high robustness and trained it with paired supervised data, achieving low-light video enhancement effects with quantitative indicators and visualization effects far exceeding those of traditional methods.

[0007] However, due to the limitations of convolution operations, current network models based on multi-scale codec structures cannot use temporal and spatial information within the global field of view to reconstruct feature information during video data processing. Furthermore, they cannot efficiently utilize spatiotemporal information to fully fuse features during feature stitching. As a result, existing methods are prone to producing significant noise, color casts, and artifacts when processing low-light videos, making the model's performance in low-light video enhancement less than ideal and leaving considerable room for improvement.

[0008] To overcome the problems existing in the current multi-scale codec structure and solve phenomena such as obvious noise, color cast, and artifacts, there is an urgent need for a low-light video enhancement method that can utilize global view spatiotemporal feature information and efficiently fuse features during feature stitching. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings and defects of existing technologies. Specifically, it aims to solve the technical problems of network models based on multi-scale codec structures being unable to reconstruct feature information using temporal and spatial information within the global field of view when processing low-light videos, and the inability to efficiently utilize spatiotemporal information to fully fuse features during feature stitching. This invention creatively proposes a low-light video enhancement method and system based on multi-scale spatiotemporal feature fusion. This invention can fully utilize the spatial internal information and inter-frame information of low-light videos to achieve high quantization metrics and superior visual sensory effects in low-light video enhancement.

[0010] The innovation of this invention lies in its first-ever approach of dividing low-light video enhancement based on multi-scale spatiotemporal feature fusion into training and application phases. In the training phase, the low-light video is preprocessed according to its different data formats within the training data of paired low-light and normal-light videos, separating different pixels into different channels using color filters. If the camera exposure settings are known, the video brightness can be initially linearly increased based on the varying degrees of darkness, with this data serving as input for the training phase. Subsequently, the low-light video enhancement algorithm generates an enhanced video based on the input low-light video and compares it with the real normal-light video, calculating the loss function and updating the parameters in the algorithm model through backpropagation. After repeatedly traversing paired data in the training dataset and repeating the above steps until a preset termination condition is met, the model structure and the updated model parameters are saved. During the usage phase, a low-light video enhancement model is first constructed based on the model structure and parameters saved during training. Then, the low-light video to be processed, after data preprocessing, is fed into the model for low-light enhancement. Finally, the enhanced video data output by the model is stored. In addition, if a real video under normal lighting conditions is available for reference, the system can use an evaluation method to measure the difference between the enhanced video and the real video under normal lighting conditions, thereby objectively and quantitatively evaluating the quality of the enhanced video and judging the effectiveness of the low-light video enhancement model. Furthermore, this invention proposes a system for implementing the above method.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] A low-light video enhancement method based on multi-scale spatiotemporal feature fusion includes two phases: training and usage.

[0013] Step 1: Training phase.

[0014] The parameter dictionary of the algorithm model is updated iteratively using a low-light video-normal-light video pairing dataset.

[0015] Specifically, step 1 includes the following steps:

[0016] Step 1.1: Preprocess the low-light video according to its data format, separate different pixels into different channels according to the color filter, and perform data preprocessing and data augmentation.

[0017] Preferably, the following processing method can be adopted:

[0018] For low-light video data, it is processed frame by frame: the low-light video is preprocessed according to its data format, so that different pixels are separated into different channels according to color filters, forming video data with a specified spatial resolution that is easy for subsequent model processing.

[0019] After video data preprocessing, data augmentation operations (including rotation, flipping, and centrosymmetric transformation) are performed on the dark light data and normal light data.

[0020] If the camera exposure settings are known, the brightness of low-light videos can be initially linearly enhanced based on the specific settings.

[0021] Step 1.2: Employ a low-light video enhancement algorithm to generate an enhanced video based on the input low-light video, compare it with the real normal light video, calculate the loss function, and update the parameters in the algorithm model through backpropagation.

[0022] Preferably, the following processing method can be adopted:

[0023] From the training dataset processed in step 1.1, a random low-light video segment with a specified number of frames is input into the algorithm model. The algorithm model performs enhancement processing on this data and outputs an enhanced video with the same number of frames. Due to the influence of data format and video data preprocessing, the width and height of the input data are either half or the same as the output data.

[0024] The enhanced video is output frame by frame.

[0025] Specifically, the low-light video enhancement algorithm model includes a codec main structure, a feature extraction and reconstruction module, and a temporal-channel attention module.

[0026] After the preprocessed low-light video data is input into the algorithm model, it first undergoes preliminary feature extraction through the encoder. The encoder consists of four layers, each with the same structure, comprising convolutional operations, skip-connection convolutional operations, downsampling convolutional operations, and activation functions for feature extraction. The feature tensors calculated by the encoder at each layer have two transmission directions: one is passed to the next layer's encoder after downsampling as its input data, and the other is passed to the feature extraction and reconstruction module for further feature processing. For this part, the parameter dictionary includes the parameters from each convolutional operation.

[0027] After initial feature extraction, features from each level are fed into the corresponding feature extraction and reconstruction module. The feature extraction and reconstruction module includes a feature embedding module, a continuous 3D Swin-Transformer block, and a feature reconstruction module. The feature embedding module segments the feature map from the encoder portion, compressing it into a feature sequence that contains global spatiotemporal information. The continuous 3DSwin-Transformer block further processes the feature sequence with global spatiotemporal information; due to the structural characteristics of the 3DSwin-Transformer block, this process does not introduce excessive additional computation. The feature reconstruction module, referencing the encoder features, reconstructs the feature sequence to a feature map of a specified size, which is then further processed in the temporal-channel attention module.

[0028] The input data for the temporal-channel attention module consists of two parts: reconstructed features from the feature extraction and reconstruction module and decoded features from a certain level of the decoder. First, the temporal-channel attention module performs global average pooling on the reconstructed and decoded features in the spatial dimension, producing vectors GAP1(FR) and GAP2(FD) with dimensions equal to the number of channels. Then, the two vectors are summed, averaged, linearly mapped, and a non-linear sigmoid function is introduced using ReLU to construct attention, resulting in attention Mask1. Next, global average pooling is performed on the reconstructed and decoded features in the temporal dimension, producing vectors GAP2(FR) and GAP2(FD) with dimensions equal to the number of frames. Again, the two vectors are summed, averaged, linearly mapped, and a non-linear sigmoid function is introduced to obtain attention Mask2. Finally, Mask1 and Mask2 attention are applied to the reconstructed features, allowing them to reflect the importance of different channels and time series in both the channel and temporal dimensions, resulting in spatiotemporally complementary features.

[0029] The decoder maintains the same overall structure as the encoder, both consisting of four different levels of convolutional operations and activation functions. The difference lies in the decoder's approach: it uses upsampling to increase spatial resolution and employs concatenated features from the previous level and a self-temporal-channel attention module within the same level as input data. After feature reconstruction across four levels, the decoder outputs a feature map of a specific dimension to a pixel shuffling layer. This layer reduces the channel dimension data of the features to one-quarter of its original size in a learnable manner and reassembles it into spatial dimensions, achieving both channel reduction and spatial dimension expansion. Finally, the output of this layer is the enhanced video data.

[0030] In the training process of the above model, a progressive training method can be adopted. Therefore, the training process of the model is divided into two steps.

[0031] Step 1: Train only the encoder-decoder part of the model. The loss function used in this process is... for:

[0032]

[0033] Among them, V NL V LL Θ1 represents the ground truth video under normal lighting conditions and the video to be processed under low lighting conditions, respectively. Θ1 represents the trainable parameters of the model. This represents a model that only contains a codec.

[0034] Step 2: Train a complete low-light enhancement model, including the main codec structure, feature extraction and reconstruction module, and temporal-channel attention module. Before training, initialize the parameter dictionary of the codec part with the results saved in Step 1. The loss function used in this process... for:

[0035]

[0036] Among them, V NL V NL Θ1 and Θ2 represent the ground truth video under normal lighting conditions and the video to be processed under low lighting conditions, respectively, and (Θ1,Θ2) are the trainable parameters of the model. This represents a complete dark light enhancement model.

[0037] Repeat step one of the training steps until the set termination condition is met, and save the network structure and model parameters; initialize the parameter dictionary of the encoder-decoder part with the result saved in step one of the training steps, repeat step two of the training steps until the set termination condition is met, and save the network structure and model parameters.

[0038] Preferably, during the training process, dark light video clips and normal light video clips are continuously and randomly input into the model until the training reaches a preset number of rounds or the evaluation index meets a preset value. At this point, the training process stops, and the model structure and model parameters are saved.

[0039] Step 2: Usage phase.

[0040] Initialize the low-light video enhancement algorithm model using the model parameters obtained in step 1, use the model to enhance the low-light video and store it.

[0041] Furthermore, if a real video under normal light conditions corresponds to the low-light video, the quality and effect of the enhanced video can be evaluated, and the effectiveness of the low-light enhancement model can be judged.

[0042] Specifically, step 2 includes the following steps:

[0043] Step 2.1: Initialize the low-light video enhancement algorithm model using the model parameters obtained in Step 1, use the model to enhance the low-light video and store it.

[0044] Preferably, the low-light video is first preprocessed according to the model input data requirements. If the camera exposure settings are known, the brightness of the low-light video is initially linearly enhanced based on the specific settings. Then, the model parameters obtained in step 1 are read, and the model is created and loaded according to the model structure. After that, the processed low-light video data is fed into the model to generate the enhanced normal-light video. Finally, the video is saved.

[0045] Step 2.2: If a real video under normal light conditions corresponds to the low-light video, evaluate the quality and effect of the enhanced video to determine the merits of the low-light enhancement model.

[0046] Preferably, in order to objectively determine the quality of the enhanced video, the objective difference between the enhanced video and the ground truth video can be calculated using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM).

[0047] Peak signal-to-noise ratio (PSNR) represents the ratio between the maximum power of a signal and the power of destructive noise. PSNR is commonly expressed in decibels (dB). A higher PSNR value indicates higher image and video quality. The calculation method for PSNR is as follows:

[0048]

[0049] Where n represents the bit depth of a single pixel in a single channel. For example, a single pixel in a grayscale image is 8 bits, a single pixel in an RGB image is also 8 bits per channel, while some raw RAW images are 14 bits.

[0050] MSE stands for Mean Squared Error, and it is calculated as follows:

[0051]

[0052] Where H and W are the height and width of the image, and I(i,j), Both represent the pixel value at position (i,j).

[0053] Structural similarity (SSIM) is used as a structural metric to compare the structural similarity between ground truth and augmented videos. Given a ground truth video frame I and an augmented video frame... SSIM is calculated as follows:

[0054]

[0055] Where, μ I It is the average value of I. yes The average value; It is the variance of I. yes variance Show I and The covariance; c1=(k1L) 2 c2 = (k2L) 2 c1 and c2 are constants used to maintain stability, and k1 and k2 are also constants used to adjust the values ​​of c1 and c2 to be much less than 1. Preferably, k1 = 0.01 and k2 = 0.03; L is the dynamic range of the pixel value, L = 2. n -1, n represents the bit depth of a single pixel in a single channel; the value of SSIM ranges from [0, 1], and the closer the value is to 1, the more similar the structures of the two video frames are.

[0056] Furthermore, based on the above methods, this invention proposes a low-light video enhancement system based on multi-scale spatiotemporal feature fusion, including a data preprocessing subsystem, a low-light enhancement model subsystem, a loss function calculation subsystem, and a result evaluation subsystem.

[0057] The data preprocessing subsystem includes a low-light video preprocessing module and a data augmentation module. The low-light video preprocessing module preprocesses the low-light video according to its data format, separating different pixels into different channels using color filters. The data augmentation module performs rotation, flipping, or centrosymmetric transformations on the low-light and normal-light videos, expanding the trainable dataset and thus enhancing the generalization ability of the low-light video enhancement model.

[0058] The low-light enhancement model subsystem comprises an encoder module, a decoder module, a feature extraction and reconstruction module, and a temporal-channel attention module. The encoder-decoder module is the core of the low-light enhancement model subsystem, consisting of four layers. Each layer has the same structure, consisting of convolutional operations, skip-connection convolutional operations, downsampling convolutional operations, and activation functions for feature extraction. This part is used for preliminary feature extraction of low-light video. Specifically, the feature extraction and reconstruction module includes a feature embedding module, a continuous 3DSwin-Transformer module, and a feature reconstruction module. Specifically, the feature extraction and reconstruction module uses the feature embedding module to segment the feature map from the encoder, compresses the feature map into a feature sequence, and then feeds it into the 3D Swin-Transformer module. The Transformer utilizes nonlocal information aggregation and global perspective for temporal-spatial information processing. The processed feature sequence is then reconstructed by the feature reconstruction module to a feature map of a specified size. The temporal-channel attention module constructs attention in the temporal and spatial dimensions, thereby obtaining the importance of different channels and time series in the channel and time dimensions by combining the reconstructed features from the feature extraction and reconstruction module and the decoded features from a certain level of the decoder module. It then fuses and splices these features in a finer-grained spatiotemporal dimension to obtain spatiotemporally complementary features.

[0059] The loss function calculation subsystem includes a codec loss function calculation module and a complete model loss function calculation module. The codec loss function calculation module is used in the first stage of the training process. It calculates the mean absolute error (MAO) between the enhanced low-light video and the real low-light video obtained through codec structure enhancement. This loss is then used to update various parameters of the codec through backpropagation and a parameter optimizer, thereby continuously optimizing the low-light enhancement performance of the codec. The complete model loss calculation module is used in the second stage of the training process. It adds a feature extraction and reconstruction module and a temporal-channel attention module to the codec structure to obtain a complete low-light video enhancement model. It calculates the MAO between the enhanced video and the real low-light video obtained from the complete low-light video enhancement model, and updates various parameters of the codec through backpropagation and a parameter optimizer, thereby continuously optimizing the low-light enhancement performance of the complete model.

[0060] The results evaluation subsystem is used to compare the enhanced video with the real normal light video, calculate objective evaluation indicators such as PSNR and SSIM, and thus quantify the quality of the low-light video enhanced by this system.

[0061] The connections between the above-mentioned components are as follows:

[0062] The data preprocessing subsystem is connected to the low-light enhancement module subsystem and the result evaluation subsystem, respectively. It provides the low-light enhancement module subsystem with low-light video and real normal light video that have been processed by video preprocessing and data augmentation and conform to the model input and output format, and provides the result evaluation subsystem with real normal light video.

[0063] In the data preprocessing subsystem, the low-light video preprocessing module and the data augmentation module are connected. The low-light video preprocessing module provides the data augmentation module with video data that meets the requirements of the data augmentation format.

[0064] The low-light enhancement model subsystem is connected to the data preprocessing subsystem, the loss function calculation subsystem, and the result evaluation subsystem, respectively. It recovers the enhanced low-light video from the low-light video input by the data preprocessing subsystem and sends it to the loss function calculation subsystem and the result evaluation subsystem.

[0065] In the low-light enhancement model subsystem, the encoder module is connected to the decoder module and the feature extraction and reconstruction module. The decoder module is connected to the encoder module and the temporal-channel attention module.

[0066] In the feature extraction and reconstruction module, the feature embedding module is connected to the 3D Swin-Transformer module; the 3DSwin-Transformer module is connected to the feature embedding module and the feature reconstruction module; and the feature reconstruction module is connected to the 3D Swin-Transformer module.

[0067] The loss function calculation subsystem is connected to the data preprocessing subsystem and the low-light enhancement model subsystem. It obtains augmented real-world normal-light video from the data preprocessing subsystem, and then obtains the codec-enhanced video and the complete model enhancement result from the low-light enhancement model subsystem. The decoder-enhanced video and the complete model enhancement result are compared with the real-world normal-light video to calculate the loss, and this loss is fed back to the low-light enhancement model subsystem to update the model parameters within the low-light enhancement model subsystem.

[0068] In the loss function calculation subsystem, the codec loss calculation module and the complete model loss calculation module work independently and are not connected to each other.

[0069] The results evaluation subsystem is connected to the data preprocessing subsystem and the low-light enhancement model subsystem, respectively. It obtains real normal lighting video from the data preprocessing subsystem and enhanced video from the low-light enhancement model subsystem. By comparing the two, it calculates evaluation indicators to quantitatively reflect the quality of the enhanced video.

[0070] The processing procedure of the above system is as follows:

[0071] Step 1: Input the low-light video into the data preprocessing subsystem for video data preprocessing and data augmentation;

[0072] Step 2: The low-light video is sent to the codec module in the low-light enhancement model subsystem. The low-light video is processed by this part to generate an enhanced video for use.

[0073] Step 3: The codec loss function calculation module in the loss function calculation subsystem receives the enhanced video generated by the codec module and the real normal lighting video generated by the data preprocessing subsystem, calculates the loss value, and sends it back to the codec module for training.

[0074] Step 4: The codec module in the dark light enhancement model subsystem calculates the loss value returned by the codec loss function calculation module in the subsystem based on the loss function, and uses the optimizer to optimize the parameters to complete one training iteration;

[0075] Step 5: Repeat steps 1 to 4 above. After each iteration through the entire training set, evaluate the model performance under the current parameters using the validation set data in the result evaluation subsystem. If the calculated quantization metric meets the preset conditions, or the current number of training iterations reaches the preset limit, stop training, save the parameter dictionary of the current encoder-decoder module, and continue to step 6.

[0076] Step 6: Load the parameter dictionary of the codec module saved in Step 5 into the codec module part of the complete model to complete the initialization of the complete model.

[0077] Step 7: Similar to Step 1, input the low-light video into the data preprocessing subsystem for preprocessing and data augmentation. The processed video data will be temporarily stored in memory.

[0078] Step 8: Send the low-light video into the complete model of the low-light enhancement model subsystem. The low-light video is processed by this part to generate an enhanced video for use.

[0079] Step 9: The loss function calculation module of the complete model in the loss function calculation subsystem receives the enhanced video generated by the complete model and the real normal lighting video generated by the data preprocessing subsystem, calculates the loss value, and sends it back to the complete model for training.

[0080] Step 10: The complete model in the dark light enhancement model subsystem calculates the loss value returned by the loss function calculation module of the complete model in the subsystem based on the loss function, and uses the optimizer to optimize the parameters to complete one training iteration;

[0081] Step 11: Repeat steps 7 to 10 above. After each iteration through the entire training set, evaluate the model performance under the current parameters using the validation set data in the result evaluation subsystem. If the calculated quantitative indicators meet the preset conditions, or the current number of training iterations reaches the preset limit, stop training, save the parameter dictionary of the current complete model, and continue to step 12.

[0082] Step 12: Initialize the complete model using the parameter dictionary saved in Step 11, input the target low-light video data into the model for low-light enhancement, save the output video, and use the result evaluation subsystem to evaluate the graphic quality of the enhanced video.

[0083] Beneficial effects

[0084] The method of the present invention has the following advantages compared with the prior art:

[0085] 1. This invention uses a multi-scale spatiotemporal feature fusion low-light video enhancement model. In the jump connection part between each level of encoder and decoder, the model is designed and introduces a feature extraction and reconstruction module and a temporal-channel attention module based on 3DSwin-Transformer.

[0086] Compared with other low-light video enhancement and denoising algorithms, the model designed in this invention can more effectively utilize encoder and decoder information to achieve higher accuracy and better performance in low-light video enhancement, especially in restoring object textures in the video, removing noise, and maintaining object boundaries between frames.

[0087] 2. This invention uses a feature extraction and reconstruction module based on 3D Swin-Transformer. In the jump connections between the encoder and decoder at various scales of the codec structure, this module captures global context information through a 3D sliding window, establishes long-range dependencies of object information in the feature map in the spatiotemporal domain, and thus expresses reconstruction features with both spatial and temporal global perspectives. This effectively reduces the differences in features between the encoder and decoder during the multi-scale information extraction and reconstruction process.

[0088] 3. This invention utilizes a temporal-channel attention module. During decoder feature concatenation, the temporal-channel attention module extracts reconstructed and decoded feature attention in both the temporal and spatial domains, and then fuses them sequentially in the feature dimension and the spatiotemporal dimension, thereby obtaining temporal-channel attention and spatiotemporally complementary features that simultaneously possess the spatiotemporal characteristics of both features. Compared to simple feature concatenation, the temporal-channel attention module can fine-tune and fuse features at a more granular level, thus achieving spatiotemporal complementarity between reconstructed and decoded features. Attached Figure Description

[0089] Figure 1 This is a flowchart of the method of the present invention.

[0090] Figure 2 This is a schematic diagram of the core algorithm model described in the present invention.

[0091] Figure 3 This is a schematic diagram of the feature extraction and reconstruction module and its sub-modules described in the method of the present invention.

[0092] Figure 4 This is a schematic diagram of the timing-channel attention module described in the method of the present invention.

[0093] Figure 5 This is a schematic diagram of the system composition of the present invention. Detailed Implementation

[0094] To better illustrate the purpose and advantages of the present invention, the method of the invention will be further described below in conjunction with the accompanying drawings and examples.

[0095] Example

[0096] This embodiment discloses a low-light video enhancement method based on multi-scale spatiotemporal feature fusion, including a training phase and an application phase.

[0097] During the training phase, in the training data of paired low-light and normal-light videos, the single-channel low-light video in Bayer RAW format is decomposed and reconstructed into four different color channels of GRBG. Furthermore, if the camera exposure settings are known, the video brightness can be initially linearly increased according to the different levels of darkness, and this data is used as input data for the training phase. Subsequently, the low-light video enhancement algorithm generates an enhanced video based on the input low-light video and compares it with the real RGB format normal-light video, calculates the loss function, and updates the parameters in the algorithm model through backpropagation. After repeatedly traversing the paired data in the training dataset and repeating the above steps until a preset termination condition is reached, the model structure and the updated model parameters are saved. In the usage phase, the low-light video enhancement model is first constructed based on the model structure and parameters saved during the training phase. Then, the low-light video to be processed is decomposed and reconstructed and used as input data to be fed into the model for low-light enhancement. Finally, the enhanced video data output by the model is stored. In addition, if there is already a real video under normal lighting conditions available for reference, the system can use evaluation methods to measure the difference between the enhanced video and the real video under normal lighting conditions, thereby objectively and quantitatively evaluating the quality of the enhanced video and judging the effectiveness of the low-light video enhancement model. The flowchart of this embodiment is as follows: Figure 1 As shown.

[0098] Previous low-light enhancement algorithms have failed to address the feature differences between encoding and decoding, and may produce pseudo-detail textures during feature splicing and fusion. Therefore, this embodiment discloses a multi-scale spatiotemporal feature fusion method for low-light video enhancement. This method combines a skip connection feature extraction and reconstruction module with a temporal-channel attention module, resulting in superior performance in restoring object textures, removing noise, and maintaining inter-frame object boundaries. A schematic diagram of the model is shown below. Figure 2 As shown.

[0099] This embodiment discloses a low-light video enhancement method based on multi-scale spatiotemporal feature fusion, which can directly enhance the input low-light video to the effect of normal light during the usage stage, without applying any prior conditions or restrictions, and without human intervention.

[0100] The details of this embodiment are as follows.

[0101] A low-light video enhancement method based on multi-scale spatiotemporal feature fusion is divided into two stages: training and application, and includes the following steps:

[0102] Step 1: During the training phase, a low-light video-normal-light video paired dataset is used to iteratively update the parameter dictionary of the algorithm model.

[0103] Step 1.1: The single-channel low-light video in Bayer RAW format is split into four different color channels of GRBG through decomposition and reconstruction. Data preprocessing such as standardization and normalization is performed on the data, and data augmentation such as image flipping and symmetry transformation is performed.

[0104] The video data is processed frame-by-frame. For each frame of Bayer RAW format image, the single-channel image is decomposed into four different GRBG channels based on the pixels acquired through different color filters, resulting in image data with a spatial resolution of one-quarter of the original but four times the number of channels. After decomposition and reconstruction, the low-light and normal-light data are randomly rotated, flipped, or subjected to centrosymmetric transformations. Furthermore, if the camera exposure settings are known, the brightness of the low-light video can be initially linearly enhanced based on the specific settings.

[0105] Step 1.2: The low-light video enhancement algorithm generates an enhanced video based on the input low-light video, compares it with the real RGB format normal light video, calculates the loss function, and updates the parameters in the algorithm model through the backpropagation process.

[0106] A short, randomly selected low-light video segment with a specified number of frames is input into the algorithm model from the training dataset processed in step 1.1. The model performs enhancement processing on this data and outputs an enhanced video with the same number of frames. Due to the decomposition and reconstruction process, the width and height of the input data are half that of the output data. The enhanced video is output frame-by-frame in RGB format, with dimensions identical to the original RAW format low-light video before decomposition and reconstruction.

[0107] In this embodiment, the low-light video enhancement algorithm model comprises three parts: the encoder / decoder main structure, the feature extraction and reconstruction module, and the temporal-channel attention module. When the decomposed and reconstructed low-light video data is input into the algorithm model, it first undergoes preliminary feature extraction through the encoder. This part is divided into four layers, each with the same structure, consisting of convolutional operations, skip-connection convolutional operations, downsampling convolutional operations, and activation functions for feature extraction. The feature tensors calculated by the encoder at each layer have two transmission directions: one is to pass them to the next level encoder after downsampling as its input data, and the other is to pass them into the feature extraction and reconstruction module for further feature processing. For this part, the parameter dictionary mainly includes the parameters from each convolutional operation.

[0108] After the initial feature extraction by the encoder, features from each level are fed into the corresponding feature extraction and reconstruction module. The key components of the feature extraction and reconstruction module are the feature embedding module, a continuous 3D Swin-Transformer block, and the feature reconstruction module. The feature embedding module segments the feature map from the encoder, compressing it into a feature sequence that includes global spatiotemporal information. The continuous 3D Swin-Transformer block further processes the feature sequence with global spatiotemporal information, and due to the structural characteristics of the 3D Swin-Transformer block, this process does not introduce excessive additional computation. The feature reconstruction module, referencing the encoder features, reconstructs the feature sequence to a feature map of a specified size. This feature map is then further processed in the next module, the temporal-channel attention module. The feature extraction and reconstruction module and its sub-modules are as follows: Figure 3 As shown.

[0109] The temporal-channel attention module receives two sets of input data: reconstructed features from the feature extraction and reconstruction module and decoded features from a specific layer of the decoder. First, the module performs global average pooling on the reconstructed and decoded features in the spatial dimension, generating vectors GAP1(FR) and GAP2(FD) with dimensions equal to the number of channels. Then, the two vectors are summed, averaged, linearly mapped, and non-linearity is introduced through ReLU. A sigmoid function is then applied to construct attention, resulting in attention Mask1. Next, global average pooling is performed on the reconstructed and decoded features in the temporal dimension, generating vectors GAP2(FR) and GAP2(FD) with dimensions equal to the number of frames. Similarly, the two vectors are summed, averaged, linearly mapped, and non-linearity is introduced through sigmoid to obtain attention Mask2. Finally, Mask1 and Mask2 attention are applied to the reconstructed features, allowing them to reflect the importance of different channels and time series in both the channel and temporal dimensions, resulting in spatiotemporally complementary features. The temporal-channel attention module is described below. Figure 4 As shown.

[0110] The decoder section maintains the same overall structure as the encoder section, both consisting of four different levels of convolutional operations and activation functions. The structural difference lies in that the decoder section in this embodiment employs upsampling to increase spatial resolution and uses decoded features from the previous level and concatenated features from the same level's temporal-channel attention module as input data. After feature reconstruction at four levels, the decoder section outputs a feature map of a specific dimension to a pixel shuffling layer. This layer reduces the channel dimension data of the features to one-quarter of its original size in a learnable manner and reorganizes it into spatial dimensions, achieving a reduction in the number of channels and an increase in spatial dimensions. Finally, the data output by this layer is the enhanced video data in RGB format.

[0111] In the training process of the above model, a progressive training method is adopted. Therefore, the training process of the model is divided into two steps.

[0112] The first step involves training only the encoder-decoder portion of the model. The loss function used in this process is:

[0113]

[0114] in, and Θ1 represents the ground truth video in RGB format under normal lighting conditions and the video to be processed in RAW format under low lighting conditions, respectively. Θ1 represents the trainable parameters of the model. This represents a model that only contains a codec.

[0115] The second step involves training a complete low-light enhancement model, including the main codec structure, feature extraction and reconstruction module, and temporal-channel attention module. Before training, the parameter dictionary for the codec is initialized with the results saved from the first training step. The loss function used in this process is...

[0116]

[0117] in, and Θ1 and Θ2 represent the ground truth video in RGB format under normal lighting conditions and the video to be processed in RAW format under low lighting conditions, respectively. Θ1 and Θ2 are the trainable parameters of the model. This represents a complete dark light enhancement model.

[0118] Step 1.3: Repeat step 1.2 of the training process until the set termination condition is met, and save the network structure and model parameters. Initialize the parameter dictionary of the encoder-decoder part to the result saved in step 1. Repeat step 2 of the training process in step 1.3 until the set termination condition is met, and save the network structure and model parameters.

[0119] Specifically, during the training process, RAW format dark light video clips and RGB format normal light video clips are continuously and randomly input into the model until the training reaches a preset number of rounds or a certain evaluation metric meets a certain preset value. At this point, the training process stops, and the model structure and model parameters are saved.

[0120] Step 2.1: Initialize the low-light video enhancement algorithm model with the model parameters obtained from Step 1, and use the model to enhance the low-light video and store it.

[0121] Specifically, the RAW format low-light video is first decomposed and reassembled into four-channel video data frame by frame. If the camera exposure settings are known, the brightness of the low-light video can be initially linearly enhanced according to the specific settings. Then, the model parameters trained in step 1 are read, and the model is created and loaded according to its structure. Next, the processed low-light video data is fed into the model to generate an enhanced RGB format normal-light video. This video is then saved locally, completing this step.

[0122] Step 2.1: If a real video under normal lighting conditions corresponds to the low-light video, the quality and effect of the enhanced video can be evaluated to determine the merits of the low-light enhancement model. The objective difference between the enhanced video and the ground truth video can be calculated using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). PSNR represents the ratio between the maximum power of the signal and the power of destructive noise. The unit of PSNR is commonly expressed in decibels (dB); a higher value indicates higher image / video quality. The calculation method for PSNR is as follows:

[0123]

[0124] Where n represents the bit depth of a single pixel in a single channel. For example, a single pixel in a grayscale image is 8 bits, a single pixel in an RGB image is also 8 bits for each channel, and some raw RAW images are 14 bits.

[0125] MSE stands for Mean Squared Error, and it is calculated as follows:

[0126]

[0127] Where H and W are the height and width of the image, and I(i,j) and This represents the pixel value at position (i,j).

[0128] Structural similarity (SSIM) is used as a structural metric to compare the structural similarity between ground truth and enhanced videos. Given a ground truth video frame I and an enhanced video frame... SSIM is calculated as follows:

[0129]

[0130] Where, μ I It is the average value of I. yes The average value; It is the variance of I. yes variance Indicate I and The covariance; c1=(k1L) 2 c2 = (k2L) 2 c1 and c2 are constants used to maintain stability, while k1 and k2 are constants used to adjust the values ​​of c1 and c2 to be much less than 1, where k1 = 0.01 and k2 = 0.03; L is the dynamic range of the pixel value, i.e., L = 2. n-1, n represents the bit depth of a single pixel in a single channel; the value of SSIM ranges from [0, 1], and the closer the value is to 1, the more similar the structures of the two video frames are.

[0131] This embodiment further discloses a low-light video enhancement system based on multi-scale spatiotemporal feature fusion, comprising the following subsystems: a data preprocessing subsystem, a low-light enhancement model subsystem, a loss function calculation subsystem, and a result evaluation subsystem.

[0132] The data preprocessing subsystem includes a low-light video decomposition and reconstruction subsystem and a data augmentation subsystem. The low-light video decomposition and reconstruction subsystem decomposes the single-channel image in the low-light video into four different GRBG channels, based on pixels acquired through different color filters, resulting in image data with a spatial resolution one-quarter of the original but four times the number of channels. The data augmentation subsystem performs rotation, flipping, or centrosymmetric transformations on the low-light and normal-light videos to expand the trainable dataset, thereby enhancing the generalization ability of the low-light video enhancement model.

[0133] The low-light enhancement model subsystem includes an encoder module, a decoder module, a feature extraction and reconstruction module, and a temporal-channel attention module. The encoder-decoder module is the core of the low-light enhancement model subsystem, consisting of four layers. Each layer has the same structure, comprising convolutional operations, skip-connection convolutional operations, downsampling convolutional operations, and activation functions for feature extraction. This part primarily performs preliminary feature extraction from low-light videos. The feature extraction and reconstruction module uses a feature embedding module to segment the feature map from the encoder, compressing it into a feature sequence. This sequence is then fed into a 3D Swin-Transformer module, which utilizes the Transformer's nonlocal information aggregation and global perspective for large-scale temporal-spatial information processing. Finally, the processed feature sequence is reconstructed by the feature reconstruction module to a feature map of a specified size. The temporal-channel attention module constructs attention in both temporal and spatial dimensions, thereby determining the importance of different channels and time series in the channel and temporal dimensions by combining the reconstructed features from the feature extraction and reconstruction module and the decoded features from a certain layer of the decoder. This results in a more granular spatiotemporal fusion and splicing of complementary spatiotemporal features.

[0134] The loss function calculation subsystem includes a codec loss function calculation module and a complete model loss function calculation module. The codec loss function calculation module is used in the first stage of the training process. It calculates the mean absolute error (MAO) between the enhanced low-light video and the real low-light video obtained through codec structure enhancement. This loss is then used to update various parameters of the codec part through backpropagation and a parameter optimizer, thereby continuously optimizing the low-light enhancement performance of the codec. The complete model loss calculation module is used in the second stage of the training process. It adds a feature extraction and reconstruction module and a temporal-channel attention module to the codec structure to obtain a complete low-light video enhancement model. It calculates the MAO between the enhanced video and the real low-light video obtained from the complete low-light video enhancement model, and updates various parameters of the codec part through backpropagation and a parameter optimizer, thereby continuously optimizing the low-light enhancement performance of the complete model.

[0135] The result evaluation subsystem is used to compare the enhanced video with the real normal light video, calculate objective evaluation indicators such as PSNR and SSIM, and thus quantify the quality of the low-light video enhanced by this system.

[0136] Figure 5 This is a schematic diagram of the above system.

[0137] The connections between the above-mentioned components are as follows:

[0138] The data preprocessing subsystem is connected to the low-light enhancement module subsystem and the result evaluation subsystem. It provides the low-light enhancement module subsystem with processed low-light video and real normal light video that conform to the model input and output format after decomposition, recombination and data augmentation, and provides the result evaluation subsystem with real normal light video.

[0139] In the data preprocessing subsystem, the low-light video preprocessing module and the data augmentation module are connected. The low-light video preprocessing module provides the data augmentation module with video data that meets the requirements of the data augmentation format.

[0140] The low-light enhancement model subsystem is connected to the data preprocessing subsystem, the loss function calculation subsystem, and the result evaluation subsystem. It recovers the enhanced low-light video from the low-light video input by the data preprocessing subsystem and sends it to the loss function calculation subsystem and the result evaluation subsystem.

[0141] In the low-light enhancement model subsystem, the encoder module is connected to the decoder module and the feature extraction and reconstruction module. The decoder module is connected to the encoder module and the temporal-channel attention module.

[0142] In the feature extraction and reconstruction module, the feature embedding module is connected to the 3D Swin-Transformer module; the 3DSwin-Transformer module is connected to the feature embedding module and the feature reconstruction module; and the feature reconstruction module is connected to the 3D Swin-Transformer module.

[0143] The loss function calculation subsystem is connected to the data preprocessing subsystem and the low-light enhancement model subsystem. It obtains augmented real-world normal-light video from the data preprocessing subsystem, and then obtains the codec-enhanced video and the complete model enhancement result from the low-light enhancement model subsystem. The decoder-enhanced video and the complete model enhancement result are compared with the real-world normal-light video to calculate the loss, and this loss is fed back to the low-light enhancement model subsystem to update the model parameters within the low-light enhancement model subsystem.

[0144] In the loss function calculation subsystem, the codec loss calculation module and the complete model loss calculation module work independently and are not connected to each other.

[0145] The results evaluation subsystem is connected to the data preprocessing subsystem and the low-light enhancement model subsystem. It obtains real normal lighting video from the data preprocessing subsystem and enhanced video from the low-light enhancement model subsystem. By comparing the two, it calculates evaluation indicators to quantitatively reflect the quality of the enhanced video.

[0146] The usage method of the above system is as follows:

[0147] Step 1: Input the low-light video into the data preprocessing subsystem for decomposition, reconstruction, and data augmentation. The processed video data will be temporarily stored in memory.

[0148] Step 2: The low-light video is sent to the codec module in the low-light enhancement model subsystem. The low-light video is processed by this part to generate an enhanced video for use.

[0149] Step 3: The codec loss function calculation module in the loss function calculation subsystem receives the enhanced video generated by the codec module and the real normal lighting video generated by the data preprocessing subsystem, calculates the loss value, and sends it back to the codec module for training.

[0150] Step 4: The codec module in the dark light enhancement model subsystem calculates the loss value returned by the codec loss function calculation module in the subsystem based on the loss function, and uses the optimizer to optimize the parameters to complete one training iteration;

[0151] Step 5: Repeat steps 1 to 4 above. After each iteration through the entire training set, evaluate the model performance under the current parameters using the validation set data in the result evaluation subsystem. If the calculated quantization metric meets the preset conditions, or the current number of training iterations reaches the preset limit, stop training, save the parameter dictionary of the current encoder-decoder module, and continue to step 6.

[0152] Step 6: Load the parameter dictionary of the codec module saved in Step 5 into the codec module part of the complete model to complete the initialization of the complete model.

[0153] Step 7: Similar to Step 1, input the low-light video into the data preprocessing subsystem for decomposition, reconstruction, and data augmentation. The processed video data will be temporarily stored in memory.

[0154] Step 8: Send the low-light video into the complete model of the low-light enhancement model subsystem. The low-light video is processed by this part to generate an enhanced video for use.

[0155] Step 9: The loss function calculation module of the complete model in the loss function calculation subsystem receives the enhanced video generated by the complete model and the real normal lighting video generated by the data preprocessing subsystem, calculates the loss value, and sends it back to the complete model for training.

[0156] Step 10: The complete model in the dark light enhancement model subsystem calculates the loss value returned by the loss function calculation module of the complete model in the subsystem based on the loss function, and uses the optimizer to optimize the parameters to complete one training iteration;

[0157] Step 11: Repeat steps 7 to 10 above. After each iteration through the entire training set, evaluate the model performance under the current parameters using the validation set data in the result evaluation subsystem. If the calculated quantitative indicators meet the preset conditions, or the current number of training iterations reaches the preset limit, stop training, save the parameter dictionary of the current complete model, and continue to step 12.

[0158] Step 12: Initialize the complete model using the parameter dictionary saved in Step 11, input the target low-light video data into the model for low-light enhancement, save the output video, and use the result evaluation subsystem to evaluate the graphic quality of the enhanced video.

[0159] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A low-light video enhancement method based on multi-scale spatiotemporal feature fusion, characterized in that, Including training and usage phases; During the training phase, the parameter dictionary of the algorithm model is iteratively updated using a low-light video-normal-light video paired dataset; First, in the training data of low-light video-normal-light video pairing, the low-light video is preprocessed according to its different data formats, so that different pixels are separated into different channels according to the color filter; if the camera exposure setting is known, the video brightness is further initially linearly increased according to the different levels of video darkness, and this data is used as the input data for the training stage. Subsequently, the low-light video enhancement algorithm generates an enhanced video based on the input low-light video, compares it with the real normal-light video, calculates the loss function, and updates the parameters in the algorithm model through backpropagation, including the following steps: A short, low-light video with a specified number of frames is randomly selected from the training dataset and input into the algorithm model. The algorithm model performs enhancement processing on this data and outputs an enhanced video with the same number of frames. Due to the influence of data format and video data preprocessing, the width and height of the input data are half of the output data, or the same as the output data. The enhanced video is output frame by frame. The low-light video enhancement algorithm model includes a codec main structure, a feature extraction and reconstruction module, and a temporal-channel attention module; After the preprocessed low-light video data is input into the algorithm model, it first undergoes preliminary feature extraction through the encoder. The encoder consists of four layers, each with the same structure, comprising convolution operations, skip connection convolution operations, downsampling convolution operations, and activation functions for feature extraction. The feature tensors calculated by the encoder at each layer have two transmission directions: one is to pass them to the next level encoder after downsampling as its input data, and the other is to pass them into the feature extraction and reconstruction module for further feature processing. For this part, the parameter dictionary includes the parameters of each convolution operation. After initial feature extraction, features at each level are fed into the feature extraction and reconstruction module at the same level. The feature extraction and reconstruction module includes a feature embedding module, a continuous 3D Swin-Transformer block, and a feature reconstruction module. The feature embedding module is used to segment the feature map from the encoder, compress the feature map into a feature sequence, and make the feature sequence contain global spatiotemporal information. The continuous 3D Swin-Transformer block is used to further process the feature sequence with global spatiotemporal information. Due to the structural characteristics of the 3D Swin-Transformer block, this process does not introduce too much additional computation. The feature reconstruction module is used to reconstruct the feature sequence to a feature map of a specified size with reference to the encoder features. This feature map is then further sent to the temporal-channel attention module for processing. The input data for the temporal-channel attention module consists of two parts: reconstructed features from the feature extraction and reconstruction module and decoded features from a certain level of the decoder. The temporal-channel attention module first performs global average pooling on the reconstructed and decoded features in the spatial dimension, generating vectors 𝐺𝐴𝑃1(𝐹𝑅) and 𝐺𝐴𝑃1(𝐹𝐷) with dimensions equal to the number of channels. Then, the two vectors are added together, averaged, linearly mapped, and a non-linear sigmoid function is introduced using ReLU to construct the attention function. 𝑀𝑎𝑠𝑘1; then, global average pooling is performed on the reconstructed features and decoded features in the time dimension to produce vectors 𝐺𝐴𝑃2(𝐹𝑅) and 𝐺𝐴𝑃2(𝐹𝐷) with dimensions equal to the number of frames; subsequently, the two vectors are added together and averaged, then linearly mapped and nonlinearity is introduced through Sigmoid to obtain attention 𝑀𝑎𝑠𝑘2; finally, 𝑀𝑎𝑠𝑘1 and 𝑀𝑎𝑠𝑘2 attention are applied to the reconstructed features to make them reflect the importance of different channels and time series in the channel dimension and time dimension, thus obtaining spatiotemporal complementary features; The decoder part is consistent with the encoder part in terms of overall structure, both consisting of four different levels of convolution operations and activation functions; the difference is that the decoder part uses upsampling to increase spatial resolution and uses the decoded features from the previous level and the concatenated features from the self-temporal-channel attention module of the same level as input data. After four levels of feature reconstruction, the decoder outputs a feature map of a specific dimension to a pixel shuffling layer. This layer reduces the channel dimension data of the features to one-quarter of its original size in a learnable manner and reorganizes it into the spatial dimension, achieving a reduction in the number of channels and an expansion of the spatial dimension. Finally, the data output by this layer is the enhanced video data. Continue to iterate through the pairs of data in the training dataset and repeat the above steps until the preset termination condition is met. During the usage phase, based on the model structure and parameters from the training phase, a low-light video enhancement model is constructed. The low-light video to be processed is preprocessed and then fed into the model as input data for low-light enhancement. Finally, the enhanced video data output by the model is stored. If there are existing real videos under normal lighting conditions for reference, the difference between the enhanced video and the real video under normal lighting conditions can be measured by evaluation methods, thereby achieving an objective and quantitative assessment of the quality of the enhanced video and judging the effectiveness of the low-light video enhancement model.

2. The low-light video enhancement method based on multi-scale spatiotemporal feature fusion as described in claim 1, characterized in that, The model training process employs a progressive training approach, including the following steps: Step 1: Train only the encoder-decoder part of the model; the loss function used in this process is... for: in, , These represent the ground truth video under normal lighting conditions and the video to be processed under low lighting conditions, respectively. These are the trainable parameters of the model. This represents a model that only contains a codec; Step 2: Train a complete dark-light enhancement model including the main codec structure, feature extraction and reconstruction module, and temporal-channel attention module; before training, initialize the parameter dictionary of the codec part with the results saved in Step 1; the loss function used in this process is... for: in, , These represent the ground truth video under normal lighting conditions and the video to be processed under low lighting conditions, respectively. These are the trainable parameters of the model. This represents a complete dark light enhancement model; Repeat the first step of training until the set termination condition is met, and save the network structure and model parameters; initialize the parameter dictionary of the encoder-decoder part with the result saved in the first step of training, repeat the second step of training until the set termination condition is met, and save the network structure and model parameters.

3. The low-light video enhancement method based on multi-scale spatiotemporal feature fusion as described in claim 1, characterized in that, During training, dark light video clips and normal light video clips are continuously and randomly input into the model until the training reaches a preset number of rounds or the evaluation metric meets a preset value. At this point, the training process stops, and the model structure and model parameters are saved.

4. The low-light video enhancement method based on multi-scale spatiotemporal feature fusion as described in claim 1, characterized in that, The implementation method during the usage phase is as follows: First, the low-light video is preprocessed according to the model input data requirements. If the camera exposure settings are known, the brightness of the low-light video is initially linearly enhanced according to the specific settings. Then, the trained model parameters are read, and the model is created and loaded according to the model structure. After that, the processed low-light video data is fed into the model to generate the enhanced normal light video. Finally, the video is saved. If a real video under normal lighting conditions corresponds to the low-light video, then evaluate the quality and effect of the enhanced video to determine the merits of the low-light enhancement model.

5. The low-light video enhancement method based on multi-scale spatiotemporal feature fusion as described in claim 4, characterized in that, The objective difference between enhanced and ground truth videos is calculated using peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). Peak signal-to-noise ratio (PSNR) represents the ratio between the maximum power of a signal and the power of destructive noise. The unit of PSNR is decibels (dB). A higher PSNR value indicates higher image and video quality. The calculation method for PSNR is as follows: Where n represents the bit depth of a single pixel in a single channel; MSE stands for Mean Squared Error, and it is calculated as follows: Where H and W are the height and width of the image, , Indicates that in the enhanced video The value of the pixel at that location; Structural similarity (SSIM) is used as a structural metric to compare the structural similarity between ground truth and augmented videos; given ground truth video frames... With enhanced video frames The SSIM is calculated as follows: in, yes The average value, yes The average value; yes variance yes variance express and covariance; , , , These are all constants used to maintain stability. , All are constants used for adjustment , The value should be much smaller than 1; 𝐿 is the dynamic range of the pixel value. , Represents the bit depth of a single pixel in a single channel. The value of SSIM ranges from [0, 1]. The closer the value is to 1, the more similar the structures of the two video frames are.

6. The low-light video enhancement method based on multi-scale spatiotemporal feature fusion as described in claim 5, characterized in that, ， 。

Images