Low-light video enhancement method and apparatus

Abstract

This invention provides a method and apparatus for enhancing low-light video, the method comprising: acquiring N consecutive video frames V to be processed. i ; V of N video frames i Any two consecutive video frames V t‑1 and V t The input is fed into the target video enhancement model to obtain the output of the target video enhancement model and the video frame V. t Corresponding enhanced video frame O t By inputting the low-light video frames to be processed into an available target video enhancement model, enhanced video frames with highly consistent temporal information are obtained. The target video enhancement model is obtained by jointly training two consecutive single-frame data as samples and combined with a Siamese enhancement model, which can improve the image quality and visual quality of the enhanced video frames output by the target video enhancement model.

Description

Technical Field

[0001] This invention belongs to the field of video processing technology, and particularly relates to a method and apparatus for enhancing low-light video. Background Technology

[0002] Due to insufficient lighting conditions in the real world and the limited performance of shooting equipment, low-light videos are ubiquitous in the real world.

[0003] Low-light videos are characterized by low brightness, noise, blur, and other degraded information, making it difficult to express the true information of the captured scene and thus limiting their usability. This objectively restricts their application scenarios. Related technologies typically utilize processing techniques such as gamma correction and Retinex methods to enhance low-light videos. However, these techniques cannot perceive the characteristics of the input video during the enhancement process, which can easily lead to over-enhancement or unnatural enhancement results. When using machine learning-based methods for video enhancement, manually designed operators cannot accurately represent the low-light video information of the real world, resulting in poor robustness of the video enhancement results. All of these problems reduce the enhancement effect of a single video frame, leading to poor overall video enhancement results. Summary of the Invention

[0004] The present invention provides a low-light video enhancement method and apparatus to solve the defect of poor robustness of low-light video enhancement in the prior art because the manually designed operators cannot accurately represent the low-light video information of the real world, thereby improving the enhancement effect of low-light video.

[0005] This invention provides a low-light video enhancement method, the method comprising:

[0006] Obtain N consecutive video frames V to be processed i , where 1≤i≤N, and i is a natural number, and N is a natural number greater than 1;

[0007] The N video frames V i Any two consecutive video frames V t-1 and V t The input is fed into the target video enhancement model to obtain the output of the target video enhancement model and the video frame V. t Corresponding enhanced video frame O t t is a natural number greater than 1;

[0008] The target video enhancement model is obtained through joint training with a Siamese enhancement model. The target video enhancement model is based on two consecutive sample video frames V. j-1 and V j As a sample, to be compared with the sample video frame V jThe corresponding sample labels are used for training, and the Siamese augmentation model is obtained by training on two consecutive sample video frames V′. j-1 and V j As a sample, to be compared with the sample video frame V j-1 The corresponding sample labels are trained to obtain the sample video frame V′. j-1 For the sample video frame V j-1 Fusion of the sample video frames V j-1 and V j The optical flow information between them is obtained, where j is a natural number greater than 1.

[0009] According to the present invention, a low-light video enhancement method is provided, wherein the sample video frame V′ j-1 Determine this through the following steps:

[0010] For the sample video frame V j-1 and V j After processing, V is obtained. j-1 and V j Optical flow diagram between

[0011] Fusion of the optical flow map With sample video frame V j The sample fusion frame V′ is obtained. j-1 .

[0012] According to a low-light video enhancement method provided by the present invention, the target video enhancement model is determined through the following steps:

[0013] Sample video frame V j-1 and V j and with V j The corresponding sample labels are input into the first pre-defined supervised model for training, and the sample video frames V′ are simultaneously input into the model. j-1 and V j and with V′ j-1 The corresponding sample labels are input into a pre-defined second supervised model for training. The first supervised model and the second supervised model are determined by the same deep network and loss function. The weights of the first supervised model and the second supervised model are shared during training.

[0014] When the loss function converges, the first supervised model after training is used as the target video enhancement model, and the second supervised model after training is used as the Siamese enhancement model.

[0015] According to a low-light video enhancement method provided by the present invention, both the first supervised model and the second supervised model are determined by the same multi-scale network, wherein the multi-scale network includes a feature encoding layer and a weight redistribution layer;

[0016] The feature encoding layer is used to obtain global features from sample video frames input to the multi-scale network, and the weight redistribution layer is used to distribute the global features to complete the weight redistribution among the multi-scale networks.

[0017] According to the low-light video enhancement method provided by the present invention, the first supervised model and the second supervised model are determined based on the temporal consistency loss function and the image quality loss function.

[0018] According to the low-light video enhancement method provided by the present invention, the loss functions of the first supervised model and the second supervised model are:

[0019]

[0020] in, It is a structural similarity loss function used to represent the similarity between different images, ω s for The coefficient; It is the perceptual quality loss function, used to calculate the distance between the supervised set and the training set in the samples, ω. p for The coefficient; It is a smoothing loss function used to represent the smoothness of the output image, ω h for The coefficient; It is the total variational loss function, used to represent the total variation of neighboring pixels, ω tv for The coefficient; It is a color loss function used to represent the degree of constraint on the colors in an image, ω. c for The coefficient; It is the temporal consistency loss function, used to represent the degree of temporal consistency between video frames, where λ is... The coefficient.

[0021] According to a low-light video enhancement method provided by the present invention, the sample video frames include low-light video data and standard video data, and the sample video frames of the target video enhancement model and the twin enhancement model are preprocessed through the following steps:

[0022] The low-light video data is augmented using random horizontal flipping, vertical flipping, and rotation operations, and the resolution of the augmented low-light video data and the standard video data is adjusted to the target range.

[0023] The present invention also provides a video enhancement device, comprising:

[0024] The video acquisition module is used to acquire N consecutive video frames V to be processed. i , where 1≤i≤N, and i is a natural number, and N is a natural number greater than 1;

[0025] The video enhancement module is used to enhance the N video frames V i Any two consecutive video frames V t-1 and V t The input is fed into the target video enhancement model to obtain the output of the target video enhancement model and the video frame V. t Corresponding enhanced video frame O t t is a natural number greater than 1;

[0026] The target video enhancement model is obtained through joint training with a Siamese enhancement model. The target video enhancement model is based on two consecutive sample video frames V. j-1 and V j As a sample, to be compared with the sample video frame V j The corresponding sample labels are used for training, and the Siamese augmentation model is obtained by training on two consecutive sample video frames V′. j-1 and V j As a sample, to be compared with the sample video frame V j-1 The corresponding sample labels are trained to obtain the sample video frame V′. j-1 For the sample video frame V j-1 Fusion of the sample video frames V j-1 and V j The optical flow information between them is obtained, where j is a natural number greater than 1.

[0027] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the low-light video enhancement methods described above.

[0028] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described low-light video enhancement methods.

[0029] This invention provides a video enhancement method, a video enhancement device, and an electronic device. By inputting a low-light video frame to be processed into an available target video enhancement model, an enhanced video frame with highly consistent temporal information is obtained. The target video enhancement model is obtained by jointly training two consecutive single-frame data as samples and combined with a Siamese enhancement model, which can improve the image quality and visual quality of the enhanced video frame output by the target video enhancement model. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating the video enhancement method provided in an embodiment of the present invention;

[0032] Figure 2 This is one of the interface diagrams of the video enhancement method provided in the embodiments of the present invention;

[0033] Figure 3 This is the second schematic diagram of the interface of the video enhancement method provided in the embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the structure of the video enhancement method apparatus provided in an embodiment of the present invention;

[0035] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0037] The following is combined with Figure 1 The low-light video enhancement method provided in the embodiments of the present invention includes:

[0038] Step 110: Obtain N consecutive video frames V to be processed. i , where 1≤i≤N, and i is a natural number, and N is a natural number greater than 1.

[0039] In this step, a frame refers to a still image. A video can consist of multiple consecutive images, meaning that a video frame contains multiple consecutive images.

[0040] It should be noted that the N consecutive video frames to be processed refer to the number of frames in the video to be enhanced. Multiple consecutive single frames can be randomly selected from the N video frames and input into the model or device used for video enhancement, and the enhanced video frames will be output.

[0041] In this embodiment, 200 video frames are obtained from the low-light video data for use in the subsequent video enhancement process.

[0042] It should be noted that low-light videos are characterized by low brightness and are filled with noise, blur, and other degraded information, making it difficult to express the true information of the scene being filmed. Relevant video processing techniques can be used to adjust the brightness or noise of the filmed scene video to obtain low-light video. For example, the brightness of the collected video can be adjusted by gamma correction, and the required low-light video data can be obtained by adding Poisson noise and Gaussian noise to the video.

[0043] Step 120: Divide N video frames V i Any two consecutive video frames V t-1 and V t The input is fed into the target video enhancement model to obtain the output of the target video enhancement model and the video frame V. t Corresponding enhanced video frame O t t is a natural number greater than 1;

[0044] The target video enhancement model is obtained through joint training with the Siamese enhancement model. The target video enhancement model is based on two consecutive sample video frames V. j-1 and V j As a sample, to be compared with sample video frame V j The corresponding sample labels are used for training. The Siamese augmentation model is based on two consecutive sample video frames V′. j-1 and V j As a sample, to be compared with sample video frame V j-1 The corresponding sample labels are trained to obtain the sample video frame V′. j-1 For sample video frame V j-1 Fusion Sample Video Frames V j-1 and V j The optical flow information between them is obtained, where j is a natural number greater than 1.

[0045] In this step, when performing video frame enhancement, one video frame and its preceding frame are selected from N videos and input into the target video enhancement model each time. After a certain number of iterations, the target video enhancement model outputs the enhanced video frame corresponding to the selected video frame. When N enhanced video frames are obtained, the current video enhancement task is considered complete.

[0046] It should be noted that the first input video frame can use the last enhanced video frame as the preceding frame. The preceding frame and the first video frame are input into the target video enhancement model to obtain the enhanced video frame O1 of the first video frame.

[0047] It should be noted that the sample label refers to the data label corresponding to the sample video frame. This label can be a numerical label, such as "0, 1, 2...", or other data information indicating the number of second samples.

[0048] In this step, the target video enhancement model is obtained through joint training with the Siamese enhancement model. This means that when the target video model and the Siamese enhancement model can redistribute and share weights during training, the feature information and corresponding network layer weights learned by the Siamese enhancement model from the input data can be passed to the target video enhancement model. At the same time, the feature information and corresponding network layer weights learned by the target video enhancement model from the input data can also be passed to the Siamese enhancement model. This allows both the target video enhancement model and the Siamese enhancement model to learn the temporal and single-frame information of the input data after training.

[0049] It should be noted that during the training phase of the target video enhancement model and the Siamese enhancement model, the two models need to be trained separately. During the training process, the weights of the two network models are set to share. Through weight sharing, the two models can learn the temporal information or single-frame information obtained from the input data. However, during the testing phase, considering the processing efficiency of video enhancement, the Siamese enhancement model can be turned off, and the video frame to be enhanced can be directly input into the target video enhancement model to output the enhanced video frame.

[0050] In this step, optical flow is the instantaneous velocity and direction of pixel motion of a spatial object on the observation imaging plane. Optical flow can represent the spatial changes of pixels between adjacent video frames, and therefore can represent the correlation between adjacent frames, or the motion trend between objects in adjacent frames. Optical flow (u,v) can be expressed as:

[0051] (u,v)=(x1,y1)-(x2,y2)

[0052] Where (u,v) is the optical flow at coordinate (x,y), (x1,y1) represents the pixel value at (x,y) in the previous frame, (x2,y2) represents the pixel value at (x,y) in the next adjacent frame, the sign of (u,v) represents the direction of motion, and the magnitude of the value represents the magnitude of the motion offset.

[0053] In this embodiment, low-light video data containing 200 video frames is first acquired. Then, the video frames of the first video frame V1 and the second video frame V2 are selected from the video frames and V2 is input into the target video enhancement model. After 100 iterations, the enhanced video frame O2 corresponding to the second video frame V2 is output.

[0054] According to the low-light video enhancement method provided in the embodiments of this application, by inputting the low-light video frame to be processed into an available target video enhancement model, an enhanced video frame with highly consistent temporal information is obtained. The target video enhancement model is obtained by jointly training two consecutive single-frame data as samples and combined with a Siamese enhancement model, which can improve the image quality and visual quality of the enhanced video frame output by the target video enhancement model.

[0055] Optionally, sample video frame V′ j-1 The following steps are used to determine: for sample video frames V j-1 and V j After processing, V is obtained. j-1 and V j Optical flow diagram between Fusion optical flow map With sample video frame V j The sample fusion frame V′ is obtained. j-1 .

[0056] In this embodiment, two consecutive preceding frames V are arbitrarily selected from the N video frames to be processed. t-1 and the current frame V t As input to the optical flow fusion module, V is used in the optical flow fusion module. t-1 and V t The optical flow map was obtained after processing. Then the optical flow map is applied to the bilinear sampling layer. With the preceding frame V t-1 The fusion process yields the fused frame V′. j-1 The fused frame contains optical flow information from the previous frame to the current frame.

[0057] It should be noted that the optical flow fusion module is used to extract the preceding frame V. t-1 and the current frame V tThe temporal information in the optical flow fusion module includes an optical flow prediction network, a bilinear sampling layer, and a channel layer. The optical flow prediction network can be the FlowNet model and its improved model (FlowNet 2.0), the SpyNet model, or the PWC-Net model based on the feature pyramid, etc.

[0058] In this embodiment, 7000 sample video frames are selected from the database, with the same number of low-light video frames as standard video frames. The first video frame V1 and the second video frame V2 of these sample video frames are input into the improved optical flow prediction network (FlowNet 2.0). This optical flow prediction network uses fixed weight parameters and does not require training. Then, optical flow prediction is performed on V1 and V2 according to the following function to obtain the corresponding optical flow map.

[0059]

[0060] Where FlowNet2.0(·) represents the FlowNet 2.0 model, and Concat(·) represents the optical flow prediction function.

[0061] Then the optical flow map is applied to the bilinear sampling layer. The fused frame V′1 is obtained by fusing V1 with V1.

[0062] According to the low-light video enhancement method provided in the embodiments of this application, the temporal information of two consecutive video frames is obtained by the optical flow fusion module, and then the optical flow map obtained by the optical flow fusion module is fused with the previous frame to obtain a fused frame containing optical flow information. The fused frame contains optical flow information from the previous frame to the current frame, which can provide high-quality temporal information for subsequent training models.

[0063] Optionally, the target video enhancement model is determined through the following steps:

[0064] Sample video frame V j-1 and V j and with V j The corresponding sample labels are input into the first pre-defined supervised model for training, and the sample video frames V′ are simultaneously input into the model. j-1 and V j and with V′ j-1 The corresponding sample labels are input into a pre-defined second supervised model for training. The first supervised model and the second supervised model are determined by the same deep network and loss function. The weights of the first supervised model and the second supervised model are shared during training. When the loss function converges, the trained first supervised model is used as the target video enhancement model, and the trained second supervised model is used as the twin enhancement model.

[0065] In this embodiment, the target video enhancement model and the Siamese enhancement model are obtained by training a first supervised model and a second supervised model, respectively. Before training, the first supervised model and the second supervised model are set to have the same network structure to form a Siamese network. During training, since the network structures of the two supervised models are the same, the weights of the two supervised models can be shared between network layers. The feature data output by the training of the two supervised models are calculated by a preset loss function to obtain the temporal information and single-frame information in the feature map. When the preset loss function converges, the training of the two supervised models ends, and the target video enhancement model is obtained.

[0066] It's important to note that when training a supervised model, the samples need to be divided into a training set and a supervision set. The training set is used to simulate the data samples for the model to learn the video information contained in the input data. The supervision set is a separate set of samples reserved during model training. It can be used to adjust the model's hyperparameters and to conduct an initial assessment of the model's capabilities. It is typically used during iterative model training to verify the current model's generalization ability (accuracy, recall, etc.) and to decide whether to stop training. The appropriate partitioning ratio is related to the amount of information the model learns during training and the degree of training. When there are too few samples used for training a deep model, overfitting is likely to occur, resulting in poor generalization ability of the model.

[0067] In this embodiment, the video enhancement model and the twin model are each composed of two identical deep networks, which can be convolutional neural networks, deep neural networks or other network models.

[0068] In this embodiment, the loss function used during the training of the first supervised model and the second supervised model is the same. This loss function can be one or a combination of several of the following: structural similarity loss function, perceptual quality loss function, smoothness loss function, total variation loss function, temporal consistency loss function, or color loss function. Specifically, the structural similarity loss function is used to represent the similarity between different images, the perceptual quality loss function is used to calculate the distance between the supervised set and the training set in the samples, the smoothness loss function is used to represent the smoothness of the output image, the total variation loss function is used to represent the total variation of adjacent pixels, the color loss function is used to represent the degree of constraint on the color in the image, and the temporal consistency loss function is used to represent the degree of temporal consistency between video frames.

[0069] exist Figure 2In the illustrated embodiment, 7000 sample video frames are selected from the database. The first video frame V1 and the second video frame V2 are directly linearly connected in the channel layer to obtain training samples. These samples are then divided into training and supervision sets in an 8:2 ratio and input into the first supervised model for training. Simultaneously, V1 and V2 are input into the improved optical flow prediction network (FlowNet 2.0). This optical flow prediction network uses fixed weight parameters and does not require training. Optical flow prediction is then performed on V1 and V2 according to the following function to obtain the corresponding optical flow map. Then the optical flow map is applied to the bilinear sampling layer. The fused frame V′1 is obtained by fusing V1 with V1. At the channel layer, the fused frame V′1 is concatenated with V1 to obtain a training sample. This sample is then divided into a training set and a supervision set in an 8:2 ratio and input into the second supervised model for training. Training ends when the preset loss function converges. The trained first supervised model outputs the enhanced sample video frame O corresponding to V2. j The second supervised model outputs the augmented sample video frame O corresponding to V1 after training. j-1 The network parameters are adjusted based on the display effect of the output enhanced sample video frames to output the enhanced sample video frames with the best display effect. At this time, the first supervised model can be used as the target video enhancement model. Figure 2 The box arrows shown indicate that the sample video frames are fed into the next model after being combined through the channel layer.

[0070] According to the low-light video enhancement method provided in the embodiments of this application, by setting the target video enhancement model and the Siamese enhancement model to the same network structure and sharing weights during training, the temporal information and single-frame information in continuous video frames can be effectively extracted, thereby obtaining a usable target video enhancement model.

[0071] Optionally, both the first supervised model and the second supervised model are determined by the same multi-scale network, which includes a feature encoding layer and a weight redistribution layer. The feature encoding layer is used to obtain global features from the sample video frames input to the multi-scale network, and the weight redistribution layer is used to distribute the global features to complete the weight redistribution between the multi-scale networks.

[0072] It should be noted that the feature encoding layer consists of two 2D convolutional functions with a kernel of 3 and a stride of 1, two 2D convolutional functions with a kernel of 1 and a stride of 1, and a PreLU activation function, used to encode features in the input data. The weight redistribution layer includes a feature compression layer and a redistribution layer. The feature compression layer consists of a global maximum pooling branch and a global average pooling branch. Except for the different global pooling strategies, the two branches are identical, consisting of the corresponding global pooling strategies, ReLU activation functions, and fully connected layers. The redistribution layer uses a broadcast mechanism to redistribute the weights of the compressed features at the channel level. Specifically, the feature compression layer uses a global pooling strategy and a fully connected layer to globally compress the extracted features, obtaining global features. This way, effective information from the original features is obtained while ineffective information is suppressed. In the redistribution layer, the compressed global features are reasonably allocated to complete the weight redistribution.

[0073] exist Figure 3 In the illustrated embodiment, the input data is first downsampled twice to increase the dimensionality of the original feature map, resulting in features with higher dimensionality. Figure 1 Simultaneously, the downsampled features are input into the feature encoding layer for encoding, resulting in the encoded features. Figure 2 Then, the upsampling operation is used to modify the features. Figure 2 Dimensionality reduction is performed, at this point, the features Figure 2 Dimensions and features Figure 1 The dimensions are the same, and the features are... Figure 2 With features Figure 1 The concatenation is performed, and the concatenated feature map is then input into the feature encoding module for re-encoding. The re-encoded feature map is then upsampled to make its dimension the same as the original feature map. Finally, the feature maps are concatenated to obtain the global feature map, which is then input into the weight redistribution module for subsequent weight redistribution.

[0074] This embodiment provides a specific construction of a multi-scale network module and the implementation process of a weight redistribution mechanism, which can obtain compressed global features from the input video features and complete weight redistribution.

[0075] According to the low-light video enhancement method provided in the embodiments of this application, the features from the input data of the multi-scale network are downsampled and upsampled, and then processed by the feature encoding layer to output global features. These global features are then input into the weight redistribution layer, and weight information is obtained based on the effective information in the global features to achieve weight redistribution between network layers.

[0076] Optionally, the loss functions for the first supervised model and the second supervised model are determined based on the temporal consistency loss function and the image quality loss function.

[0077] In this embodiment, the image quality loss function is used to evaluate the quality of an image. It can be one or a combination of several of the following: structural similarity loss function, perceptual quality loss function, smoothness loss function, total variation loss function, or color loss function. Specifically, the structural similarity loss function represents the similarity between different images, the perceptual quality loss function calculates the distance between the supervision set and the training set in the samples, the smoothness loss function represents the smoothness of the output image, the total variation loss function represents the total variation of adjacent pixels, the color loss function represents the degree of constraint on the colors in the image, and the temporal consistency loss function represents the degree of temporal consistency between video frames.

[0078] It should be noted that the similarity loss function The expression is as follows:

[0079]

[0080] Where, μ x and μ y It is the average of the input and output images after they are read into a [0,255] digital image. and It is the variance, σ xy C1 and C2 are constants to prevent division by zero, set to 0.001. The structural similarity loss function can constrain the structural similarity of images during training. When it converges, it can effectively ensure that the texture features of the output image are similar to the features of the supervision set, thus ensuring the quality of texture details.

[0081] Perceived quality loss function l p The expression is as follows:

[0082]

[0083] Where, φ ij This represents the j-th convolutional layer in the i-th layer module using the VGG16 network, where ||·|1 represents the L1 loss function, i.e., the mean absolute error. pred It is the model's output image, V gt These are the corresponding supervised set images. The perceptual quality loss function can constrain training at the high-dimensional semantic level of the image, and when it converges, it can effectively guarantee the visual quality of the output image.

[0084] Smoothing loss function The expression is as follows:

[0085]

[0086]

[0087] in, Representing Kronecker product, M * δ represents the secondary mask, and δ is the parameter controlling the sensitivity of the algorithm, set to 1. H and W are the length and width of the video frame.

[0088] Total variational loss function The expression is as follows:

[0089]

[0090] Among them, V pred (i,j) represents the specific value in the i-th row and j-th column of the output image.

[0091] Color loss function The expression is as follows:

[0092]

[0093] Among them, ∠(·,·) uses RGB colors as 3D vectors to calculate the angle between the two colors. When it converges, it can effectively ensure that the colors of the output image are more consistent with the real world and have realistic visual quality.

[0094] Temporal consistency loss function The expression is as follows:

[0095]

[0096] in, This represents a multi-scale network module.

[0097] In this embodiment, the loss functions of the first supervised model and the second supervised model are designed as a combination of the temporal consistency loss function and the image quality loss function. In this way, when the feature maps output by the two supervised models are calculated using this combination function, temporal information and single-frame information can be extracted from the feature maps.

[0098] According to the low-light video enhancement method provided in the embodiments of this application, a new loss function is obtained by combining a loss function used to represent the degree of temporal consistency between video frames and a loss function used to represent the evaluation of image quality. This new loss function is used in the training process of the first supervised model and the second supervised model, and can extract the required temporal information and single-frame information from the samples input to the model, respectively.

[0099] Optionally, the loss functions for the first supervised model and the second supervised model are:

[0100]

[0101] in, It is a structural similarity loss function used to represent the similarity between different images, ω s for The coefficient; It is the perceptual quality loss function, used to calculate the distance between the supervised set and the training set in the samples, ω. p for The coefficient; It is a smoothing loss function used to represent the smoothness of the output image, ω h for The coefficient; It is the total variational loss function, used to represent the total variation of neighboring pixels, ω tv for The coefficient; It is a color loss function used to represent the degree of constraint on the colors in an image, ω. c for The coefficient; It is the temporal consistency loss function, used to represent the degree of temporal consistency between video frames, where λ is... The coefficient.

[0102] In this embodiment, the loss functions of the first supervised model and the second supervised model... The loss function is determined by multiple loss functions and is used to calculate the temporal and single-frame information in the sample video frames input to the supervised model. The corresponding coefficients of each loss function can be adaptively adjusted according to the training effect of the first and second supervised models.

[0103] In this embodiment, ω is taken. s =ω p =ω h =ω c =1, ω tv =0.001, λ=20, the corresponding loss function is:

[0104]

[0105] In this embodiment, when training the first supervised model and the second supervised model, Adam is used as the optimizer to optimize the gradient. The learning rate is initially set to 0.0005, and the learning rate is decreased once every 50 iterations at a rate of 1.2. A total of 500 iterations of training are performed. When the loss function converges during training, it indicates that the first supervised model and the second supervised model have been trained successfully, and the trained first supervised model can be used as the target video enhancement model.

[0106] According to the low-light video enhancement method provided in the embodiments of this application, by using a loss function Perceptual loss function Color loss function TV loss function and Huber loss function To construct the loss function And determine the corresponding coefficients of each loss function, and then use this loss function... It can calculate the temporal and single-frame information contained in the sample video frames input into the model, and then use this information to improve the texture detail quality, visual quality, and temporal consistency quality of the video frames.

[0107] Optionally, the sample video frames include low-light video data and standard video data. The sample video frames of the target video enhancement model and the twin enhancement model are preprocessed through the following steps: the low-light video data is augmented using random horizontal flip, vertical flip and rotation operations, and the resolution of the augmented low-light video data and standard video data is adjusted to the target range.

[0108] It should be noted that the sample video frames include paired low-light-normal-brightness videos with a total length of more than 6000 frames to ensure sufficient and diverse training data.

[0109] In this embodiment, after obtaining a large number of normal brightness videos, a synthesis method is used to create paired low-light-standard brightness video data, and sample video frames are obtained.

[0110] In this embodiment, geometric transformations such as random horizontal flipping, vertical flipping, and rotation are used to augment the low-light video data. While maintaining data alignment, the data is combined with the original standard video data to form new sample video frame data. Finally, the resolution of the new sample video frame is adjusted to 1920×1080.

[0111] In some embodiments, the object of data preprocessing may be a sample video frame input to the optical flow fusion module or a sample video frame input to the supervised learning model.

[0112] The low-light video enhancement method provided in the embodiments of this application can enrich the distribution of training set data and improve the generalization performance and robustness of the trained supervised model by augmenting the input continuous frame data or the sample data input into the supervised model for training. It can also prevent the deep model from overfitting due to insufficient sample data during training to a certain extent.

[0113] Combination Figure 4 The low-light video enhancement device provided in the embodiments of the present invention will be described below. The low-light video enhancement device described below and the low-light video enhancement method described above can be referred to and correspond to each other.

[0114] This invention provides a low-light video enhancement device, the device comprising:

[0115] Video acquisition module 410 is used to acquire N consecutive video frames V to be processed. i , where 1≤i≤N, and i is a natural number, and N is a natural number greater than 1;

[0116] Video enhancement module 420 is used to enhance N video frames V i Any two consecutive video frames V t-1 and V t The input is fed into the target video enhancement model to obtain the output of the target video enhancement model and the video frame V. t Corresponding enhanced video frame O t t is a natural number greater than 1;

[0117] The target video enhancement model is obtained through joint training with the Siamese enhancement model. The target video enhancement model is based on two consecutive sample video frames V. j-1 and V j As a sample, to be compared with sample video frame V j The corresponding sample labels are used for training. The Siamese augmentation model is based on two consecutive sample video frames V′. j-1 and V j As a sample, to be compared with sample video frame V j-1 The corresponding sample labels are trained to obtain the sample video frame V′. j-1 For sample video frame V j-1 Fusion Sample Video Frames V j-1 and V j The optical flow information between them is obtained, where j is a natural number greater than 1.

[0118] According to the low-light video device provided in the embodiments of this application, the video acquisition module 410 acquires multiple consecutive video frames to be processed, and the video enhancement module 420 inputs the low-light video frames to be processed into an available target video enhancement model to obtain enhanced video frames with highly consistent temporal information. The target video enhancement model is obtained by jointly training two consecutive single-frame data as samples and combined with a Siamese enhancement model, which can improve the image quality and visual quality of the enhanced video frames output by the target video enhancement model.

[0119] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communication interface 520, a memory 630, and a communication bus 540. The processor 510, communication interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a low-light video enhancement method, which includes: acquiring N consecutive video frames V to be processed. i Where 1 ≤ i ≤ N, and i is a natural number, and N is a natural number greater than 1; N video frames V i Any two consecutive video frames V t-1 and V t The input is fed into the target video enhancement model to obtain the output of the target video enhancement model and the video frame V. t Corresponding enhanced video frame O t t is a natural number greater than 1; where the target video enhancement model is obtained through joint training with the Siamese enhancement model, and the target video enhancement model is based on two consecutive sample video frames V j-1 and V j As a sample, to be compared with sample video frame V j The corresponding sample labels are used for training. The Siamese augmentation model is based on two consecutive sample video frames V′. j-1 and V j As a sample, to be compared with sample video frame V j-1 The corresponding sample labels are trained to obtain the sample video frame V′. j-1 For sample video frame V j-1 Fusion Sample Video Frames V j-1 and V j The optical flow information between them is obtained, where j is a natural number greater than 1.

[0120] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] The present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform a low-light video enhancement method provided by the methods described above, the method comprising: acquiring N consecutive video frames V to be processed. i Where 1 ≤ i ≤ N, and i is a natural number, and N is a natural number greater than 1; N video frames V i Any two consecutive video frames V t-1 and V t The input is fed into the target video enhancement model to obtain the output of the target video enhancement model and the video frame V. t Corresponding enhanced video frame O t t is a natural number greater than 1; where the target video enhancement model is obtained through joint training with the Siamese enhancement model, and the target video enhancement model is based on two consecutive sample video frames V j-1 and V j As a sample, to be compared with sample video frame V j The corresponding sample labels are used for training. The Siamese augmentation model is based on two consecutive sample video frames V′. j-1 and V j As a sample, to be compared with sample video frame V j-1 The corresponding sample labels are trained to obtain the sample video frame V′. j-1 For sample video frame V j-1 Fusion Sample Video Frames V j-1 and V j The optical flow information between them is obtained, where j is a natural number greater than 1.

[0122] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of dim light video enhancement, characterized by, The method comprises: Obtaining continuous N video frames V to be processed i Wherein, 1≤i≤N, and i is a natural number, N is a natural number greater than 1; inputting the N video frames V i and any two continuous video frames V t-1 and V t to a target video enhancement model to obtain an enhanced video frame O t corresponding to the video frame V t , t is a natural number greater than 1. The target video enhancement model is obtained through joint training with a Siamese enhancement model. The target video enhancement model is based on two consecutive sample video frames V. j-1 and V j As a sample, to be compared with the sample video frame V j The corresponding sample labels are used for training, and the Siamese augmentation model is obtained by training on two consecutive sample video frames V′. j-1 and V j As a sample, to be compared with the sample video frame V j-1 The corresponding sample labels are trained to obtain the sample video frame V′. j-1 For the sample video frame V j-1 Fusion of the sample video frames V j-1 and V j The optical flow information between them is obtained, where j is a natural number greater than 1; The target video enhancement model is obtained by joint training with the twin enhancement model, which means that when the target video model and the twin enhancement model are trained, the twin enhancement model transmits the feature information and the corresponding network layer weight learned from the input data to the target video enhancement model through weight redistribution and sharing, and the target video enhancement model transmits the feature information and the corresponding network layer weight learned from the input data to the twin enhancement model, so that the target video enhancement model and the twin enhancement model both learn the time sequence information and single frame information of the input data after training; the sample video frame V' j-1 by the steps of: For the sample video frame V j-1 and V j After processing, V is obtained. j-1 and V j Optical flow diagram between fusing the optical flow map with the sample video frame V j , resulting in a sample fused frame V' j-1 ; N video frames to be processed are taken as two time-sequentially continuous pre-frame V t-1 and current frame V t as the input of the optical flow fusion module, in which V t-1 and V t are processed to obtain an optical flow map Then the optical flow map is fused with the pre-frame V t-1 to obtain a fused frame V′ j-1 which contains the optical flow information from the previous frame to the current frame; the optical flow fusion module is used to extract the time-sequential information in the pre-frame V t-1 and the current frame V t , and the optical flow fusion module comprises an optical flow prediction network, a bilinear sampling layer and a channel layer.

2. The dim video enhancement method of claim 1, wherein, The target video enhancement model is determined by the following steps: sample video frames V j-1 and V j and sample labels corresponding to V j are input into a preset first supervised model for training, while sample video frames V' j-1 and V j and sample labels corresponding to V' j-1 are input into a preset second supervised model for training, wherein the first supervised model and the second supervised model are determined by the same deep network and loss function; the weights of the first supervised model and the second supervised model during training are shared weights. When the loss function converges, the trained first supervised model is used as the target video enhancement model, and the trained second supervised model is used as the twin enhancement model.

3. The dim video enhancement method of claim 2, wherein, The first supervised model and the second supervised model are both determined by the same multi-scale network, and the multi-scale network comprises a feature encoding layer and a weight redistribution layer. The feature encoding layer is used to obtain global features from the sample video frames input into the multi-scale network, and the weight redistribution layer is used to distribute the global features to complete the weight redistribution between the multi-scale networks.

4. The dim video enhancement method of claim 2, wherein, The loss function of the first supervised model and the second supervised model is determined based on the time sequence consistency loss function and the image quality loss function.

5. The dim video enhancement method of claim 4, wherein, The loss function of the first supervised model and the second supervised model is: wherein, is a structural similarity loss function, used to represent the similarity between different images, ω s is a coefficient of ; is a perceptual quality loss function, used to calculate the distance between the supervised set and the training set in the sample, ω p is a coefficient of ; is a smoothness loss function, used to represent the smoothness of the output image, ω h is a coefficient of ; is a total variation loss function, used to represent the total variation of adjacent pixels, ω tv is a coefficient of ; is a color loss function, used to represent the degree of constraint on the color in the image, ω c is a coefficient of ; is a temporal consistency loss function, used to represent the degree of temporal consistency between video frames, λ is a coefficient of .

6. The dim video enhancement method according to any one of claims 1-5, characterized in that, The sample video frames include dark light video data and standard video data, and the sample video frames of the target video enhancement model and the twin enhancement model are preprocessed by the following steps: The dark light video data is expanded by random horizontal flipping, vertical flipping and rotation operation, and the resolution of the expanded dark light video data and standard video data is adjusted to the target range.

7. A dark-light video enhancement apparatus applying the dark-light video enhancement method according to claim 1, characterized by, The device comprises: The video acquisition module is configured to acquire a continuous N video frames V to be processed i wherein 1≤i≤N, i is a natural number, and N is a natural number greater than 1. a video enhancement module, configured to input the N video frames V i any two continuous video frames V t-1 and V t into a target video enhancement model, to obtain an enhanced video frame O t corresponding to the video frame V t t is a natural number greater than 1; The target video enhancement model is obtained through joint training with a Siamese enhancement model. The target video enhancement model is based on two consecutive sample video frames V. j-1 and V j As a sample, to be compared with the sample video frame V j The corresponding sample labels are used for training, and the Siamese augmentation model is obtained by training on two consecutive sample video frames V′. j-1 and V j As a sample, to be compared with the sample video frame V j-1 The corresponding sample labels are trained to obtain the sample video frame V′. j-1 For the sample video frame V j-1 Fusion of the sample video frames V j-1 and V j The optical flow information between them is obtained, where j is a natural number greater than 1.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the dark light video enhancement method according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the dark light video enhancement method according to any one of claims 1 to 6.

Images