A low-light video enhancement method and system

Abstract

The application discloses a low-illumination video enhancement method and system, and belongs to the technical field of image processing. The method comprises the following steps: acquiring initial videos of different categories, extracting a normal-light video frame sequence, and generating a low-illumination video sequence through the normal-light video sequence; taking the two sequences as a group of contrast video frame sequences, constructing a low-illumination video dataset containing multiple groups of contrast video frame sequences, and dividing the low-illumination video dataset into a training set, a verification set and a test set according to a preset proportion; combining a self-attention mechanism and a convolution principle to construct a low-illumination video enhancement network model; adopting a twin network structure to train the low-illumination video enhancement network model, wherein a consistency loss, a reconstruction loss, an SSIM loss and a color loss are adopted as loss functions during the training; and inputting low-illumination video frame sequences in the test set into the low-illumination video enhancement network model obtained after the training to obtain enhanced video frame sequences.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a low-light video enhancement method and system. Background Technology

[0002] Low-light video enhancement refers to improving the visibility and quality of videos shot in low-light conditions through image processing techniques and algorithms. In low-light environments, insufficient light can lead to problems such as shadows, noise, and low contrast, making it difficult to discern image details. The goal of low-light video enhancement is to improve these issues, making the video brighter, clearer, and highlighting details. Low-light video enhancement has applications in many fields, including nighttime surveillance, autonomous driving, and computational photography. By improving the quality of low-light video, image recognition and analysis capabilities can be enhanced, improving system performance and reliability.

[0003] In recent years, deep learning methods have made significant progress in low-light video enhancement. By using deep neural networks, the mapping relationship between low-light images and normal-light images can be learned, generating more visible images. However, existing low-light video enhancement techniques are limited by the size and diversity of the datasets used for training. There are not enough pairs of low-light videos with uneven exposure and normal-light videos, making it difficult for the network to learn the mapping relationship from uneven-exposure low-light videos to normal-light videos. Moreover, existing techniques mainly focus on enhancing low-light videos with uniform illumination, and there are no network models designed for low-light videos with uneven exposure. When processing videos with uneven exposure, overexposure areas may be over-enhanced, resulting in loss of detail in overexposed areas.

[0004] Due to limitations in the size and diversity of datasets used for training in existing technologies, there are insufficient pairs of low-light videos with uneven exposure and normal lighting, making it difficult for networks to learn the mapping relationship from low-light videos with uneven exposure to normal lighting videos. Furthermore, there is a lack of network models specifically designed for low-light videos containing uneven exposure, which may lead to over-enhancing of overexposed areas and loss of detail when processing such videos. Summary of the Invention

[0005] This invention aims to address the problems existing in low-light video enhancement technologies. Existing technologies suffer from limitations in the size and diversity of training datasets, lacking sufficient pairs of low-light videos with uneven exposure and normal lighting, making it difficult for the network to learn the mapping relationship from uneven-exposure low-light videos to normal lighting videos. Furthermore, there is a lack of network models specifically designed for low-light videos containing uneven exposure, which may lead to over-enhancing of overexposed areas and loss of detail when processing such videos. To address these problems, this invention provides a low-light video enhancement method and system.

[0006] First aspect

[0007] This invention provides a low-light video enhancement method, applied to a low-light video enhancement system, comprising:

[0008] S101: Obtain initial videos of different categories, extract normal light video frame sequences from the initial videos, and generate low-light video sequences using the normal light video sequences, wherein the categories include scene, time, and exposure intensity;

[0009] S102: Take the normal light video frame sequence and the low light video frame sequence corresponding to the normal light video frame sequence as a set of comparison video frame sequences, construct a low light video dataset containing multiple sets of comparison video frame sequences, and divide the low light video dataset into training set, validation set and test set according to a preset ratio.

[0010] S103: Combining self-attention mechanism and convolution principle, a low-light video enhancement network model is constructed. The low-light video enhancement network model includes a first convolutional layer, a second convolutional layer, a third convolutional layer with a stride of 2, an adaptive feature extraction module, and a depth_to_space layer. The adaptive feature extraction module includes a first adaptive feature extraction module to a seventh adaptive feature extraction module connected in sequence. Each adaptive feature extraction module consists of a local feature extraction module, a self-attention module, and a feature fusion module connected in sequence. At the same time, a residual connection is established between the local feature extraction module and the feature fusion module. The first adaptive feature extraction module and the fifth adaptive feature extraction module, the second adaptive feature extraction module and the sixth adaptive feature extraction module, and the third adaptive feature extraction module and the seventh adaptive feature extraction module are all skip connections. The third convolutional layer is connected to the local feature extraction module of the first adaptive feature extraction module, and the feature fusion module of the seventh adaptive feature extraction module is connected to the depth_to_space layer.

[0011] S104: Select a comparison video frame sequence from the training set, adopt a Siamese network structure, and input the N-1, N, N+1 and N-2, N, N+2 frames of the low-light video frame sequence as input images to two identical low-light video enhancement network models with shared weights. Train the low-light video enhancement network models. During training, the Nth frame of the normal light video frame sequence is used as the standard video frame. The loss function used during training is consistency loss, reconstruction loss, SSIM loss and color loss.

[0012] S105: Input the contrast video frame sequence from the test set into the low-light video enhancement network model obtained after training to obtain the enhanced video frame sequence.

[0013] Second aspect

[0014] This invention provides a low-light video enhancement system, comprising:

[0015] The acquisition module is used to acquire initial videos of different categories, extract normal light video frame sequences from the initial videos, and generate low-light video sequences from the normal light video sequences. The categories include scene, time, and exposure intensity.

[0016] The first construction module is used to construct a low-light video dataset containing multiple sets of comparison video frame sequences by taking the normal light video frame sequence and the low-light video frame sequence corresponding to the normal light video frame sequence as a set of comparison video frame sequences, and dividing the low-light video dataset into a training set, a validation set and a test set according to a preset ratio.

[0017] The second construction module is used to construct a low-light video enhancement network model by combining the self-attention mechanism and the convolution principle. The low-light video enhancement network model includes a first convolutional layer, a second convolutional layer, a third convolutional layer with a stride of 2, an adaptive feature extraction module, and a depth_to_space layer. The adaptive feature extraction module includes a first adaptive feature extraction module to a seventh adaptive feature extraction module connected in sequence. Each adaptive feature extraction module is composed of a local feature extraction module, a self-attention module, and a feature fusion module connected in sequence. At the same time, a residual connection is established between the local feature extraction module and the feature fusion module. The first adaptive feature extraction module and the fifth adaptive feature extraction module, the second adaptive feature extraction module and the sixth adaptive feature extraction module, and the third adaptive feature extraction module and the seventh adaptive feature extraction module are all skip connections. The third convolutional layer is connected to the local feature extraction module of the first adaptive feature extraction module, and the feature fusion module of the seventh adaptive feature extraction module is connected to the depth_to_space layer.

[0018] The training module is used to select low-light video frame sequences from the training set. It adopts a Siamese network structure and inputs the N-1, N, N+1 and N-2, N, N+2 frames of the low-light video frame sequence as input images into two identical low-light video enhancement network models to train the low-light video enhancement network models. The Nth frame of the normal light video frame sequence is used as the standard video frame during training. The loss function used during training is consistency loss, reconstruction loss, SSIM loss and color loss.

[0019] The input module is used to input the contrast video frame sequence from the test set into the low-light video enhancement network model after training, so as to obtain the enhanced video frame sequence.

[0020] Compared with the prior art, the present invention has at least the following beneficial effects:

[0021] In this invention, different types of initial videos are collected, and normal light video frame sequences and corresponding low-light video frame sequences are extracted to form a low-light video dataset comprising multiple scenes, different times, and different exposure intensities. This large-scale dataset enhances the training effect of the constructed low-light video enhancement network model, improving video enhancement capabilities, reducing the probability of overfitting or underfitting when processing new input data, and enhancing the model's generalization ability. Furthermore, the construction of the low-light video enhancement network model combines self-attention mechanisms and convolution principles. The output feature map of each self-attention module and the feature map of the local feature extraction module of the previous layer are concatenated along the channel direction and used as the input to the feature fusion module, enhancing the network's backpropagation ability to better guide the network in low-light video enhancement. By using self-attention mechanisms between blocks, the contextual relationships and long-range dependencies between different regions are captured, thereby better representing image features. The self-attention mechanism guides the network to enhance low-light areas of the low-light image, preventing over-enhancement of overexposed areas, preserving more complete details of the initial video, and improving the video enhancement effect. Attached Figure Description

[0022] The preferred embodiments will now be described in a clear and easy-to-understand manner, in conjunction with the accompanying drawings, to further explain the above-mentioned characteristics, technical features, advantages, and implementation methods of the present invention.

[0023] Figure 1 This is a flowchart illustrating a low-light video enhancement method provided by the present invention;

[0024] Figure 2 This is a schematic diagram of the structure of a low-light video enhancement network model provided by the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of a feature fusion module provided by the present invention;

[0026] Figure 4 This is a schematic diagram of a twin network structure provided by the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of a low-light video enhancement system provided by the present invention. Detailed Implementation

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0029] To keep the drawings concise, each figure only schematically shows the parts relevant to the invention, and these do not represent the actual structure of the product. Furthermore, to facilitate understanding, in some figures, only one of components with the same structure or function is schematically depicted, or only one is labeled. In this document, "one" not only means "only one," but can also mean "more than one."

[0030] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0031] In this document, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0032] Furthermore, in the description of this invention, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0033] In one embodiment, refer to the appendix to the specification. Figure 1 The present invention provides a flowchart of a low-light video enhancement method.

[0034] This invention provides a low-light video enhancement method, applied to a low-light video enhancement system, comprising:

[0035] S101: Obtain initial videos of different categories, extract normal light video frame sequences from the initial videos, and generate low-light video sequences. The categories include scene, time, and exposure intensity.

[0036] It should be noted that, to ensure the scale and diversity of the dataset, the dataset of this invention consists of videos with uniform exposure and those containing overexposed areas, captured from different scenes and at different times. Scenes include outdoor locations such as roads, parks, and schools, as well as indoor scenes such as apartments and classrooms; times include different seasons and different times of day, both morning and afternoon.

[0037] In one possible implementation, S101 specifically includes:

[0038] S1011: Capture different types of initial videos and convert them into corresponding initial video frame sequences.

[0039] S1012: Extract multiple video frames from the initial video frame sequence and calculate the average pixel brightness and color entropy of each video frame.

[0040] S1013: Select a sequence of normal light video frames from the original video that meet the preset brightness threshold and preset color entropy threshold.

[0041] S1014: Adjust the size of the normal optical video frame sequence.

[0042] S1015: Perform data augmentation processing on the decomposed video frame sequence to obtain a first preset number of video frame sequences, wherein each video frame sequence includes a second preset number of frames.

[0043] S1016: Darken the video frame sequence obtained after data enhancement to obtain a low-light video frame sequence corresponding to the video frame sequence.

[0044] In one possible implementation, S1016 specifically includes:

[0045] S1016A: Determine the overexposure threshold of a video frame sequence containing an overexposure area using an adaptive thresholding method;

[0046] S1016B: Generate a mask for the overexposed area based on the overexposed area threshold, and use the mask to extract the overexposed area of ​​each video frame;

[0047] S1016C: The overexposure degree of the overexposure area is reduced by the adaptive histogram equalization method. Finally, the processed overexposure area is reassigned back to the original video frame to reduce its overexposure effect, and a processed video frame sequence containing the overexposure area is obtained.

[0048] S1016D: Add Gaussian and salt-and-pepper noise to simulate the noise in the low-light video, and simultaneously darken the processed video frame sequence containing overexposed areas and the uniformly exposed video frame sequence through gamma transformation.

[0049] It should be noted that those skilled in the art can set the size of the first preset quantity and the second preset quantity themselves, and no limitation is made here.

[0050] It should be noted that bright, color-rich videos are selected from the initial video sources. Specifically, video frames are randomly sampled from each video, and the average pixel brightness and color entropy of each frame are calculated. Appropriate brightness and color entropy thresholds are then set to select normal light video frame sequences that meet both brightness and color richness requirements. The size of the selected video frame sequences is adjusted to meet the output requirements of the network model. To improve the model's generalization ability, data augmentation is performed on the video frame sequences through flipping and random rotation.

[0051] The final number of video frame sequences obtained is no less than 1000, with each video frame sequence containing no less than 120 frames. The data is processed to obtain low-light video frame sequences paired with each normal-light video frame sequence. First, video frame sequences containing overexposed areas are processed to reduce the overexposure level, serving as normal-light video frame sequences for these overexposed areas. The processing of overexposed video frame sequences includes determining a threshold for the overexposed areas using an adaptive thresholding method, generating a mask for the overexposed areas based on the threshold, extracting the overexposed areas of each video frame using the mask, reducing the overexposure level of the overexposed areas using an adaptive histogram equalization method from the OpenCV library, and finally reassigning the processed overexposed areas back to the original video frames to weaken their overexposure effect. The processed video frame sequences containing overexposed areas and the uniformly exposed video frame sequences are then combined for image brightness dimming. Specifically, gamma transformation is used to reduce the brightness of each frame, and Gaussian and salt-and-pepper noise are added to simulate noise in low-light videos. The dimmed video frame sequences are then used as the low-light video frame sequences.

[0052] S102: Take the normal light video frame sequence and the low light video frame sequence corresponding to the normal light video frame sequence as a set of comparison video frame sequences, construct a low light video dataset containing multiple sets of comparison video frame sequences, and divide the low light video dataset into training set, validation set and test set according to a preset ratio.

[0053] It should be noted that this low-light video dataset is divided into training, validation, and test sets according to a predetermined ratio to use different data when training and evaluating the low-light video enhancement network model. The training set is used to train the model's parameters, the validation set is used to tune the model's hyperparameters and monitor its performance, and the test set is used to finally evaluate the model's performance.

[0054] In practical use, the validation set itself does not participate in the training process that changes the model weights. Instead, after the model has completed a training cycle using the training set, the validation set is used to evaluate the model's performance. Generally, training is stopped when the loss on the training set decreases while the loss on the validation set remains unchanged, in order to prevent overfitting.

[0055] In one possible implementation, S102 specifically includes:

[0056] S1021: Pair the adjusted low-light video frame sequence with the normal light video sequence to obtain the low-light video enhancement dataset.

[0057] S1022: Divide the low-light video enhancement dataset into training, validation, and test sets in a ratio of 7:2:1.

[0058] Reference Figure 2 Figure 1 shows a schematic diagram of the structure of a low-light video enhancement network model provided by the present invention.

[0059] Depend on Figure 2 As can be seen, the network as a whole is an encoder-decoder structure, mainly composed of convolutional layers, a local feature extraction module, a self-attention module, a feature fusion module, downsampling, upsampling, and a depth-to-space layer. The input video frame first passes through three convolutional layers to extract shallow local features such as contours and textures. Each convolutional layer is followed by a batch normalization layer and a ReLU layer. The stride of the last convolutional layer is set to 2 to achieve the downsampling effect. The feature map output from the convolutional layers serves as the input to the local feature extraction module, which consists of a residual connection module and a convolutional layer to extract local features of the image. The feature extraction module, self-attention module, and feature fusion module form an adaptive feature extraction module. In the encoder, after the adaptive feature extraction module, a convolutional layer with a stride of 2 is used for downsampling. The downsampled feature map serves as the input to the next adaptive feature extraction module, and this downsampling operation is repeated twice more, for a total of four downsampling operations in the encoder. The decoder uses a deconvolution layer after the adaptive feature extraction module for upsampling, reshaping the feature map size. This upsampling is repeated three times, and then a depth-to-space layer restores the feature map to the size of the input image. The adaptive feature extraction modules before and after the decoder are connected in skip connections, allowing the decoder stage to simultaneously contain both deep features and shallow features extracted by the encoder stage.

[0060] Reference Figure 3 The diagram shows a structural schematic of a feature fusion module provided by the present invention.

[0061] Feature fusion module, such as Figure 3As shown, it consists of one layer normalization, three 1×1 modules and two residual connection modules. The 1×1 convolution is used to fuse feature information in the channel direction, and the residual module is used to deepen the network depth and enhance the network's backpropagation ability. The local and global features extracted by the previous convolution module and self-attention module are used as input to this module for feature fusion through residual connection, so as to better guide the network to enhance low-light video.

[0062] S103: Combining self-attention mechanism and convolution principle, a low-light video enhancement network model is constructed. The low-light video enhancement network model includes a first convolutional layer, a second convolutional layer, a third convolutional layer with a stride of 2, an adaptive feature extraction module, and a depth_to_space layer. The adaptive feature extraction module includes a first adaptive feature extraction module to a seventh adaptive feature extraction module connected in sequence. Each adaptive feature extraction module consists of a local feature extraction module, a self-attention module, and a feature fusion module connected in sequence. The first and fifth adaptive feature extraction modules, the second and sixth adaptive feature extraction modules, and the third and seventh adaptive feature extraction modules are all skip connections. The third convolutional layer is connected to the local feature extraction module of the first adaptive feature extraction module, and the feature fusion module of the seventh adaptive feature extraction module is connected to the depth_to_space layer.

[0063] It should be noted that the first convolutional layer is used to perform initial feature extraction on the input low-light video frames. The second convolutional layer is used to further extract higher-level features based on the first convolutional layer. The third convolutional layer reduces the size of the feature map by using a convolution operation with a stride of 2. Each adaptive feature extraction module consists of a local feature extraction module, a self-attention module, and a feature fusion module connected in sequence. The Depth_to_space layer is used to perform upsampling and transform the number of channels in the feature map into a form more suitable for the output.

[0064] Self-attention, a variant of attention mechanisms, learns the correlations and weight distributions within the input data to adaptively adjust feature importance in low-light video enhancement tasks. By combining self-attention and convolution principles, a low-light video enhancement network model is constructed. This model aims to extract and fuse local features from the video, adaptively adjust feature weights, and use the final enhanced features for low-light video reconstruction and enhancement. The constructed low-light video enhancement network model, incorporating both self-attention and convolution principles, can be used in subsequent steps to enhance low-light videos. The structure and mechanism of this model can improve the effectiveness and quality of low-light video enhancement.

[0065] It should be noted that the output of the local feature extraction module serves as the input to the self-attention module, which employs a lightweight Transformer. When applied to the visual domain, the Transformer divides the image into blocks and uses a self-attention mechanism between these blocks to capture the contextual relationships and long-range dependencies between different regions, thereby better representing image features. The self-attention mechanism involves calculating a weight vector by evaluating the similarity between the query matrix Q and all key matrices K, and then multiplying this weight vector by the value matrix V to obtain the final representation. Matrix Q, K, and V are three learnable matrices. The lightweight Transformer uses a depthwise convolution with a k×k kernel and a stride of k instead of a regular convolution when calculating the key matrix K and the value matrix V, reducing computational complexity by decreasing the size of the key matrix K and the value matrix V. This invention learns the long-range dependencies between pixels within an image by adding a self-attention module and guides the network to enhance low-light regions of low-light images through the self-attention mechanism, while preventing over-enhancing of overexposed areas.

[0066] In one possible implementation, the feature fusion module is composed of a normalization layer, a first 1×1 convolution module, a first convolution residual module, a second 1×1 convolution module, a second convolution residual module, and a third 1×1 convolution module connected in sequence.

[0067] Reference Figure 4 The diagram shows a schematic of a twin network structure provided by the present invention.

[0068] This invention employs a twin network structure, selecting a set of sequences from a video sequence as input to train two network models that combine attention mechanisms and convolution with the same weights. The temporal consistency of the output video is enhanced by using a consistency loss function between the images output by the two network models that combine attention mechanisms and convolution, thereby reducing video flicker and artifacts.

[0069] S104: Select a low-light video frame sequence from the training set. Using a Siamese network structure, input the N-1, N, N+1 and N-2, N, N+2 frames from the low-light video frame sequence as input images to two identical low-light video enhancement network models with shared weights. Train the low-light video enhancement network models. Use the Nth frame of the normal light video frame sequence as the standard video frame during training. The loss function used during training is consistency loss, reconstruction loss, SSIM loss, and color loss.

[0070] Understandably, training a low-light video enhancement network model allows the model to understand the features and structure of low-light videos and learn how to generate better enhanced outputs from low-light inputs. In practical application, this solution implements the proposed method on PyTorch. The network is trained for a total of 2500 epochs using the AdamW optimizer with a batch size of 2. The CosineAnnealingWarmRestarts algorithm is employed, whose main principle is to gradually decrease the learning rate during training for better convergence and good generalization performance. This algorithm includes learning rate adjustment, warm restart, and epoch length adjustment. The learning rate adjustment uses a cosine function to adjust the learning rate and restart a new epoch. In this method, the initial learning rate is set to 10. -3 The cycle length is 5, and the cycle length multiplication factor after a hot restart is 2.

[0071] Among these, consistency loss measures the degree of consistency between the generated enhanced image and the target image, i.e., whether they are visually similar or identical. Reconstruction loss measures the reconstruction error between the generated enhanced image and the original low-light image, i.e., whether the enhanced image can accurately restore the details and structure of the original image. SSIM (Structural Similarity Metric) assesses the structural similarity between the generated enhanced image and the target image, including brightness, contrast, and structural information. Color loss measures the color accuracy and consistency between the generated enhanced image and the target image. These loss metrics are crucial for guiding model optimization and improvement. By calculating these loss metrics, the performance of the low-light video enhancement network model can be quantified, providing feedback signals for subsequent training. By minimizing these losses, the model can progressively improve the quality of the generated images, making them closer to the target image, thereby achieving better low-light video enhancement results.

[0072] It should be noted that this invention uses reconstruction loss to reduce the differences between output images during training, uses SSIM loss to reduce the differences between output images and label images in terms of brightness, contrast and structure, and uses color loss to reduce the color differences between output images and label images. This results in videos that are brighter, clearer and have more detail.

[0073] Specifically, consistency loss L c Specifically:

[0074]

[0075] Reconstruction loss L r Specifically:

[0076] L r =||I j -Ik ||1+||φ l (I j )-φ l (I k )‖1

[0077] Among them I n I represents the image output by the network. t Represents the label image, φ l This represents the features of the l-th layer of the VGG19 network model;

[0078] SSIM loss L ssim Specifically:

[0079]

[0080] Where N represents the image patch size, μ represents the mean, σ represents the variance, and C is a constant;

[0081] Color loss function L color Specifically:

[0082]

[0083]

[0084]

[0085] Among them, X g Y g G represents the image obtained after Gaussian blurring. (p,q) Let θ represent the Gaussian blur operator. m,n and ω m,n Represents a constant.

[0086] S105: Input the contrast video frame sequence from the test set into the low-light video enhancement network model obtained after training to obtain the enhanced video frame sequence.

[0087] It should be noted that existing methods cannot handle unevenly exposed videos due to two main reasons: insufficient dataset size (lacking enough pairs of low-light and normal-light videos for training) and the lack of a network model specifically designed for low-light videos with uneven exposure. Therefore, a low-light video enhancement method is proposed. This method constructs a dataset containing overexposed videos and videos whose overexposed effects have been manually removed, and establishes a neural network model combining self-attention and convolution. This allows for the acquisition of enhanced video frame sequences that meet the specified conditions.

[0088] Compared with the prior art, the present invention has at least the following beneficial effects:

[0089] In this invention, different types of initial videos are collected, and normal light video frame sequences and corresponding low-light video frame sequences are extracted to form a low-light video dataset comprising multiple scenes, different times, and different exposure intensities. This large-scale dataset enhances the training effect of the constructed low-light video enhancement network model, improving video enhancement capabilities, reducing the probability of overfitting or underfitting when processing new input data, and enhancing the model's generalization ability. Furthermore, the construction of the low-light video enhancement network model combines self-attention mechanisms and convolution principles. The output feature map of each self-attention module and the feature map of the local feature extraction module of the previous layer are concatenated along the channel direction and used as the input to the feature fusion module, enhancing the network's backpropagation ability to better guide the network in low-light video enhancement. By using self-attention mechanisms between blocks, the contextual relationships and long-range dependencies between different regions are captured, thereby better representing image features. The self-attention mechanism guides the network to enhance low-light areas of the low-light image, preventing over-enhancement of overexposed areas, preserving more complete details of the initial video, and improving the video enhancement effect.

[0090] Example 2

[0091] In one embodiment, refer to the appendix to the specification. Figure 2 The present invention provides a schematic diagram of the structure of a low-light video enhancement system.

[0092] The present invention provides a low-light video enhancement system 20, comprising:

[0093] The acquisition module 201 is used to acquire initial videos of different categories, extract normal light video frame sequences from the initial videos, and generate low light video frame sequences corresponding to the normal light video frame sequences. The categories include scene, time, and exposure intensity.

[0094] The first construction module 202 is used to construct a low-light video dataset containing multiple sets of comparison video frame sequences by taking the normal light video frame sequence and the low-light video frame sequence corresponding to the normal light video frame sequence as a set of comparison video frame sequences, and dividing the low-light video dataset into a training set, a validation set and a test set according to a preset ratio.

[0095] The second construction module 203 is used to construct a low-light video enhancement network model by combining the self-attention mechanism and the convolution principle. The low-light video enhancement network model includes a first convolutional layer, a second convolutional layer, a third convolutional layer with a stride of 2, an adaptive feature extraction module, and a depth_to_space layer. The adaptive feature extraction module includes a first adaptive feature extraction module to a seventh adaptive feature extraction module connected in sequence. Each adaptive feature extraction module is composed of a local feature extraction module, a self-attention module, and a feature fusion module connected in sequence. The first adaptive feature extraction module and the fifth adaptive feature extraction module, the second adaptive feature extraction module and the sixth adaptive feature extraction module, and the third adaptive feature extraction module and the seventh adaptive feature extraction module are all skip connections. The third convolutional layer is connected to the local feature extraction module of the first adaptive feature extraction module, and the feature fusion module of the seventh adaptive feature extraction module is connected to the depth_to_space layer.

[0096] Training module 204 is used to select low-light frequency frame sequences from the training set. Using a Siamese network structure, the N-1, N, N+1 and N-2, N, N+2 frames of the low-light frequency frame sequences are sequentially input as input images to two identical low-light video enhancement network models with shared weights. The low-light video enhancement network models are trained. The Nth frame of the normal light video frame sequence is used as the standard video frame during training. The loss function used during training is consistency loss, reconstruction loss, SSIM loss and color loss.

[0097] The input module 205 is used to input the contrast video frame sequence from the test set into the low-light video enhancement network model after training, so as to obtain the enhanced video frame sequence.

[0098] In one possible implementation, the acquisition module 201 specifically includes:

[0099] The shooting submodule is used to shoot different types of initial videos and convert them into initial video frames;

[0100] The calculation submodule is used to extract multiple video frames from the initial video frame and calculate the average pixel brightness and color entropy of each video frame.

[0101] The selection submodule is used to select a sequence of normal light video frames from the original video that meet the preset brightness threshold and preset color entropy threshold.

[0102] The first adjustment submodule adjusts the size of the normal optical video frame sequence;

[0103] The processing submodule is used to perform data augmentation processing on the decomposed video frame sequence to obtain a first preset number of video frame sequences, wherein each video frame sequence includes a second preset number of frames.

[0104] The acquisition submodule is used to darken the video frame sequence obtained after data augmentation and to acquire the low-light video frame sequence corresponding to the video frame sequence.

[0105] In one possible implementation, the acquisition submodule specifically includes:

[0106] The first processing unit is used to determine the overexposure threshold of a video frame sequence containing an overexposure region using an adaptive thresholding method.

[0107] The second processing unit is used to generate a mask for the overexposed area and to extract the overexposed area of ​​each video frame using the mask.

[0108] The third processing unit is used to reduce the overexposure of the overexposure area through the adaptive histogram equalization method. Finally, the processed overexposure area is reassigned back to the original video frame to reduce its overexposure effect, resulting in a processed video frame sequence containing the overexposure area.

[0109] The fourth processing unit is used to simultaneously darken the processed video frame sequence containing overexposed areas and the uniformly exposed video frame sequence through gamma transformation, and add Gaussian and salt-and-pepper noise to simulate noise in low-light videos.

[0110] In one possible implementation, the first construction module 202 specifically includes:

[0111] The pairing submodule is used to pair the adjusted low-light video frame sequence with the normal light video sequence to obtain the low-light video enhancement dataset;

[0112] The sub-module is used to divide the low-light video enhancement dataset into training, validation, and test sets in a 7:2:1 ratio.

[0113] In one possible implementation, the feature fusion module is composed of a normalization layer, a first 1×1 convolution module, a first convolution residual module, a second 1×1 convolution module, a second convolution residual module, and a third 1×1 convolution module connected in sequence.

[0114] The low-light video enhancement system 20 provided by the present invention can implement the various processes implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0115] The virtual system provided by this invention can be a system, or a component, integrated circuit, or chip in a terminal.

[0116] Compared with the prior art, the present invention has at least the following beneficial effects:

[0117] In this invention, different types of initial videos are collected, and normal light video frame sequences and corresponding low-light video frame sequences are extracted to form a low-light video dataset comprising multiple scenes, different times, and different exposure intensities. This large-scale dataset enhances the training effect of the constructed low-light video enhancement network model, improving video enhancement capabilities, reducing the probability of overfitting or underfitting when processing new input data, and enhancing the model's generalization ability. Furthermore, the construction of the low-light video enhancement network model combines self-attention mechanisms and convolution principles. The output feature map of each self-attention module and the feature map of the local feature extraction module of the previous layer are concatenated along the channel direction and used as the input to the feature fusion module, enhancing the network's backpropagation ability to better guide the network in low-light video enhancement. By using self-attention mechanisms between blocks, the contextual relationships and long-range dependencies between different regions are captured, thereby better representing image features. The self-attention mechanism guides the network to enhance low-light areas of the low-light image, preventing over-enhancement of overexposed areas, preserving more complete details of the initial video, and improving the video enhancement effect.

[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A low-light video enhancement method applied to a low-light video enhancement system, characterized in that, The method comprises the steps of: S101: obtaining initial videos of different categories, extracting a normal light video frame sequence from the initial videos, and generating a low-light video frame sequence through the normal light video frame sequence, wherein the categories include scenes, times, and exposure intensities; S102: constructing a low-light video dataset containing multiple sets of contrast video frame sequences by taking the normal light video frame sequence and the low-light video frame sequence corresponding to the normal light video frame sequence as a set of contrast video frame sequences, and dividing the low-light video dataset into a training set, a validation set, and a test set according to a preset ratio; S103: constructing a low-light video enhancement network model by combining a self-attention mechanism and a convolution principle, wherein the low-light video enhancement network model comprises a first convolution layer, a second convolution layer, a third convolution layer with a step of 2, an adaptive feature extraction module, and a depth_to_space layer, wherein the adaptive feature extraction module comprises first to seventh adaptive feature extraction modules connected in sequence, each adaptive feature extraction module is composed of a local feature extraction module, a self-attention module, and a feature fusion module connected in sequence, and a residual connection is established between the local feature extraction module and the feature fusion module, wherein the first adaptive feature extraction module and the fifth adaptive feature extraction module, the second adaptive feature extraction module and the sixth adaptive feature extraction module, and the third adaptive feature extraction module and the seventh adaptive feature extraction module are all jump connections, the third convolution layer is connected with the local feature extraction module of the first adaptive feature extraction module, and the feature fusion module of the seventh adaptive feature extraction module is connected with the depth_to_space layer; S104: selecting contrast video frame sequences from the training set, using a twin network structure, inputting N-1, N, N+1 and N-2, N, N+2 frames of images in the low-light video frame sequence as input images into two low-light video enhancement network models with the same structure and shared weights, training the low-light video enhancement network model, using the Nth frame of the normal light video frame sequence as a standard video frame, and using consistency loss, reconstruction loss, SSIM loss, and color loss as the loss function during training; S105: inputting the contrast video frame sequences in the test set into the trained low-light video enhancement network model to obtain an enhanced video frame sequence.

2. The low-light video enhancement method of claim 1, wherein, The S101 specifically comprises: S1011: shooting the initial videos of different categories and converting them into corresponding initial video frame sequences; S1012: extracting multiple video frames from the initial video frame sequences, and calculating the average values of the pixel brightness and color entropy of each video frame; S1013: selecting normal light video frame sequences meeting preset brightness threshold values and preset color entropy threshold values from the initial video frame sequences; S1014: adjusting the size of the normal light video frame sequence; S1015: The video frame sequence obtained after adjustment is subjected to data enhancement processing to obtain a first preset number of video frame sequences, wherein each of the video frame sequences comprises a second preset number of frame numbers; S1016: The video frame sequence obtained after data enhancement is subjected to dimming processing to obtain a low-illumination video frame sequence corresponding to the video frame sequence.

3. The low-light video enhancement method of claim 2, wherein, The S1016 specifically comprises: S1016A: The overexposure area threshold of the video frame sequence containing the overexposure area is determined by an adaptive threshold method; S1016B: A mask of the overexposure area is generated according to the overexposure area threshold, and the overexposure area of each video frame is extracted using the mask; S1016C: The overexposure degree of the overexposure area is reduced by an adaptive histogram equalization method, and finally the processed overexposure area is re-assigned back to the original video frame to weaken the overexposure effect, thereby obtaining a processed video frame sequence containing the overexposure area; S1016D: Gaussian and salt and pepper noise are added to simulate the noise in the low-illumination video, and gamma transformation is performed on the processed video frame sequence containing the overexposure area and the uniformly exposed video frame sequence to dim them.

4. The low-light video enhancement method of claim 1, wherein, The S102 specifically comprises: S1021: The adjusted low-illumination video frame sequence and the normal light video frame sequence are paired to obtain a low-illumination video enhancement dataset; S1022: The low-illumination video enhancement dataset is divided into the training set, the validation set and the test set in a ratio of 7:2:

1.

5. The low-light video enhancement method of claim 1, wherein, The feature fusion module is composed of a normalization layer, a first 1x1 convolution module, a first convolution residual module, a second 1x1 convolution module, a second convolution residual module and a third 1x1 convolution module connected in sequence.

6. A low-light video enhancement system characterized by, It comprises: An acquisition module is configured to acquire initial videos of different categories, extract normal light video frame sequences from the initial videos, and generate low-illumination video frame sequences corresponding to the normal light video frame sequences, wherein the categories include scenes, times, and exposure intensities; A first construction module is configured to construct a low-illumination video dataset containing multiple sets of contrast video frame sequences by taking the normal light video frame sequences and the low-illumination video frame sequences corresponding to the normal light video frame sequences as a set of contrast video frame sequences, and divide the low-illumination video dataset into a training set, a validation set and a test set according to a preset ratio. The second construction module is used for combining the self-attention mechanism and the convolution principle to construct a low-illumination video enhancement network model, wherein the low-illumination video enhancement network model comprises a first convolution layer, a second convolution layer, a third convolution layer with a step of 2, an adaptive feature extraction module and a depth_to_space layer, wherein the adaptive feature extraction module comprises first to seventh adaptive feature extraction modules connected in sequence, each of the adaptive feature extraction modules is composed of a local feature extraction module, a self-attention module and a feature fusion module connected in sequence, and a residual connection is established between the local feature extraction module and the feature fusion module, wherein the first adaptive feature extraction module and the fifth adaptive feature extraction module, the second adaptive feature extraction module and the sixth adaptive feature extraction module, and the third adaptive feature extraction module and the seventh adaptive feature extraction module are all jump connections, the third convolution layer is connected with the local feature extraction module of the first adaptive feature extraction module, and the feature fusion module of the seventh adaptive feature extraction module is connected with the depth_to_space layer; The training module is used for selecting the low-illumination video frame sequence from the training set, using a twin network structure, inputting the same image in the low-illumination video frame sequence as an input image into two identical low-illumination video enhancement network models respectively, training the low-illumination video enhancement network model, and using consistency loss, reconstruction loss, SSIM loss and color loss as loss functions during training; The input module is used for inputting the contrast video frame sequence in the test set into the low-illumination video enhancement network model obtained after training to obtain an enhanced video frame sequence.

7. The low-light video enhancement system of claim 6, wherein, The acquisition module specifically comprises: A shooting submodule is configured to shoot different categories of the initial video and convert the initial video into initial video sequence frames; A calculation submodule is configured to extract a plurality of video frames from the initial video sequence frames, calculate the pixel brightness and color entropy of each video frame, and average the pixel brightness and color entropy; A selection submodule is configured to select normal light video frame sequences that meet preset brightness threshold and preset color entropy threshold from the initial video; A first adjustment submodule is configured to adjust the size of the normal light video frame sequences; A processing submodule is configured to perform data enhancement processing on the decomposed video frame sequences to obtain a first preset number of video frame sequences, wherein each video frame sequence includes a second preset number of frames; An acquisition submodule is configured to perform dimming processing on the video frame sequences obtained after data enhancement to obtain low-illumination video frame sequences corresponding to the video frame sequences.

8. The low-light video enhancement system of claim 7, wherein, The acquisition submodule specifically comprises: A first processing unit is configured to determine an overexposure area threshold of the video frame sequence containing the overexposure area by using an adaptive threshold method; A second processing unit is configured to generate a mask of the overexposure area and extract the overexposure area of each video frame using the mask. The third processing unit is configured to reduce the overexposure degree of the overexposed area by using an adaptive histogram equalization method, and finally reassign the processed overexposed area to the original video frame to weaken the overexposure effect, thereby obtaining a processed video frame sequence containing the overexposed area. The fourth processing unit is configured to simultaneously dim the processed video frame sequence containing the overexposed area and the uniformly exposed video frame sequence by using gamma transformation, and add Gaussian and salt and pepper noise to simulate the noise in the low-illumination video.

9. The low-light video enhancement system of claim 6, wherein, The first construction module specifically comprises: A pairing submodule is configured to pair the adjusted low-illumination video frame sequence and the normal-light video frame sequence to obtain a low-illumination video enhancement dataset. A division submodule is configured to divide the low-illumination video enhancement dataset into the training set, the verification set and the test set according to a ratio of 7:2:

1.

10. The low-light video enhancement system of claim 6, wherein, The feature fusion module is composed of a normalization layer, a first 1*1 convolution module, a first convolution residual module, a second 1*1 convolution module, a second convolution residual module and a third 1*1 convolution module connected in sequence.

Images