Dark light video enhancement method and device and electronic equipment

Abstract

The invention discloses a dark-light video enhancement method and device and electronic equipment, and relates to the field of image processing, and the method comprises the steps: obtaining a to-be-enhanced dark-light video frame and an adjacent frame of the to-be-enhanced dark-light video frame; inputting a to-be-enhanced dark light video frame and an adjacent frame into the trained dark light video enhancement model; determining a first video frame feature and a target difference image feature of the to-be-enhanced dark-light video frame and the adjacent frame by using a dark-light video enhancement model; and determining a target video frame feature according to the first video frame feature and the target differential image feature, and performing dark light enhancement processing on the target video frame feature to obtain a target video frame after dark light enhancement. According to the embodiment of the invention, the difference image can provide more high-frequency information for dark light enhancement, and by combining the first video frame feature and the target difference image feature, details can be recovered more effectively, the definition of the video is improved, and the blurring problem caused by noise is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a dark-light video enhancement method and device and electronic equipment. BACKGROUND

[0002] Dark-light video enhancement is an important task in the field of computer vision, aiming to optimize the quality of videos taken in low-light environments. When shooting in weak light or at night, the camera may be affected by limited light, resulting in significant problems such as dark video pictures, noise, and lack of detail performance. In order to overcome these problems, researchers have proposed various dark-light video enhancement methods.

[0003] At present, most dark-light video enhancement methods use an end-to-end network to learn the mapping of dark-light videos to normal-light videos. However, due to the severe noise in dark-light videos, the enhanced videos often have the problem of blurred details, reducing the video clarity. SUMMARY

[0004] Therefore, the present application provides a dark-light video enhancement method, device and electronic equipment, which mainly aims to solve the problem that the current dark-light video enhancement method uses an end-to-end network to learn the mapping of dark-light videos to normal-light videos, due to the severe noise in dark-light videos, the enhanced videos often have the problem of blurred details, reducing the video clarity.

[0005] According to a first aspect of the present application, a dark-light video enhancement method is provided, comprising:

[0006] obtaining a dark-light video frame to be enhanced and a neighboring frame of the dark-light video frame to be enhanced;

[0007] inputting the dark-light video frame to be enhanced and the neighboring frame into a trained dark-light video enhancement model;

[0008] determining first video frame features and target difference image features of the dark-light video frame to be enhanced and the neighboring frame using the dark-light video enhancement model;

[0009] determining target video frame features according to the first video frame features and the target difference image features, and performing dark-light enhancement processing on the target video frame features to obtain dark-light enhanced target video frames.

[0010] According to a second aspect of the present application, a dark-light video enhancement device is provided, comprising:

[0011] an acquisition module configured to obtain a dark-light video frame to be enhanced and a neighboring frame of the dark-light video frame to be enhanced;

[0012] The input module is configured to input the dark-light video frame to be enhanced and the adjacent frame into the trained dark-light video enhancement model.

[0013] The determination module is configured to determine first video frame features and target difference image features of the dark-light video frame to be enhanced and the adjacent frame by using the dark-light video enhancement model.

[0014] The processing module is configured to determine target video frame features according to the first video frame features and the target difference image features, and perform dark-light enhancement processing on the target video frame features to obtain the dark-light enhanced target video frame.

[0015] According to a third aspect of the present application, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the dark-light video enhancement method of the first aspect is implemented.

[0016] According to a fourth aspect of the present application, an electronic device is provided, which includes a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor. When the processor executes the computer program, the dark-light video enhancement method of the first aspect is implemented.

[0017] According to a fifth aspect of the present application, a chip is provided, which includes one or more interface circuits and one or more processors. The interface circuit is configured to receive a signal from a memory of an electronic device and send the signal to the processor. The signal includes computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device performs the dark-light video enhancement method of the first aspect.

[0018] According to the above technical solutions, the dark-light video enhancement method, device and electronic device provided by the present application can obtain a dark-light video frame to be enhanced and an adjacent frame of the dark-light video frame to be enhanced. The dark-light video frame to be enhanced and the adjacent frame are input into a trained dark-light video enhancement model. The first video frame features and the target difference image features of the dark-light video frame to be enhanced and the adjacent frame are determined by using the dark-light video enhancement model. The target video frame features are determined according to the first video frame features and the target difference image features, and the target video frame features are subjected to dark-light enhancement processing to obtain the dark-light enhanced target video frame. For the disclosed embodiments, the difference image can provide more high-frequency information for dark-light enhancement. By combining the first video frame features and the target difference image features, details can be more effectively restored, the clarity of the video can be improved, and the blurring problem caused by noise can be reduced.

[0019] The above description is only a summary of the technical solutions of the present application. In order to enable the technical means of the present application to be better understood and implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present application more obvious and understandable, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, those skilled in the art can obtain other drawings from these drawings without any creative effort.

[0022] Figure 1 A flowchart of a dark-light video enhancement method provided by an embodiment of the present disclosure is shown in the figure.

[0023] Figure 2 A flowchart of another dark-light video enhancement method provided by an embodiment of the present disclosure is shown in the figure.

[0024] Figure 3 A structural diagram of a dark-light video enhancement device provided by an embodiment of the present disclosure is shown in the figure.

[0025] Figure 4 A general structural diagram of a dark-light video enhancement model provided by an embodiment of the present disclosure is shown in the figure.

[0026] Figure 5 A structural diagram of another dark-light video enhancement device provided by an embodiment of the present disclosure is shown in the figure. DETAILED DESCRIPTION

[0027] The exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of the present disclosure, which are helpful for understanding. They should be considered only as exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, the description below omits the description of well-known functions and structures. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0028] The dark-light video enhancement method, device and electronic equipment of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0029] To solve the problem that the existing technology has serious noise due to dark light video, so that the enhanced video often has fuzzy details, and the problem of reducing the video definition.

[0030] The embodiment provides a dark light video enhancement method, as shown in the method, the method comprises the steps of: Figure 1

[0031] Step 101, obtaining a dark light video frame to be enhanced and a neighboring frame of the dark light video frame to be enhanced.

[0032] In video processing, the dark light video frame to be enhanced can be a video frame shot under low light conditions, which usually has problems of much noise, fuzzy details, color distortion and the like due to insufficient light, and the visual quality thereof can be improved through dark light enhancement technology;

[0033] The neighboring frame can be a frame continuous in time with the dark light video frame to be enhanced. For example, in a video sequence, the dark light video frame to be enhanced is the Nth frame, and then the neighboring frame can be the (N-1)th frame (namely, the previous frame) and the (N+1)th frame (namely, the next frame).

[0034] In actual application, under dark light conditions, a single video frame can not provide sufficient details due to insufficient light, and the neighboring frame can provide more abundant information, which can help to repair or enhance the content of the current dark light video frame to improve the quality of the current dark light video frame.

[0035] Step 102, inputting the dark light video frame to be enhanced and the neighboring frame into a dark light video enhancement model trained.

[0036] The dark light video enhancement model can be used to increase the brightness of the dark light video frame to be enhanced, remove or reduce noise caused by low light conditions, improve the definition of the dark light video, and enhance the detail information in the dark light video, so as to avoid loss of details caused by excessive smoothing.

[0037] By inputting the dark light video frame to be enhanced and the neighboring frame into the dark light video enhancement model trained, the brightness of the dark light video frame to be enhanced can be increased, the definition of the dark light video can be improved, and the detail information in the dark light video can be enhanced.

[0038] Step 103, determining first video frame features of the dark light video frame to be enhanced and the neighboring frame and target difference image features by using the dark light video enhancement model.

[0039] The first video frame features can be video frame features after feature alignment and fusion processing are performed on second video frame features of the dark light video frame to be enhanced and third video frame features of the neighboring frame;

[0040] The difference image can be determined by comparing the difference between the dark light video frame to be enhanced and the neighboring frame.​

[0041] The target differential image feature can be information of changes between the to-be-enhanced dark-light video frame and the adjacent frame extracted from the differential image.

[0042] For the embodiments of the present disclosure, the second video frame feature of the to-be-enhanced dark-light video frame can be extracted, and the third video frame feature of the adjacent frame can be extracted. The dark-light video enhancement model can be used to perform feature alignment fusion processing on the second video frame feature and the third video frame feature to obtain the first video frame feature. The dark-light video enhancement model can be used to calculate the target differential image feature by comparing the difference between the second video frame feature and the third video frame feature.

[0043] In step 104, the target video frame feature is determined according to the first video frame feature and the target differential image feature, and dark-light enhancement processing is performed on the target video frame feature to obtain a dark-light enhanced target video frame.

[0044] The target video frame feature can be a fusion video frame feature obtained by performing feature fusion processing on the first video frame feature and the target differential image feature.

[0045] The target video frame can be a clear video frame after dark-light enhancement processing.

[0046] For the embodiments of the present disclosure, the target video frame feature can be determined according to the first video frame feature and the target differential image feature, and the dark-light enhancement module in the dark-light video enhancement model can be used to perform dark-light enhancement processing on the target video frame feature. The dark-light enhanced target video frame feature map is input into the up-sampling module in the dark-light video enhancement model to obtain a target video frame under normal lighting.

[0047] The dark-light enhancement module can be a residual network module, which can include a Reflection padding layer, a convolution layer, a Relu activation function, a convolution layer, and a skip connection between the input and the output.

[0048] The up-sampling module can include three up-sampling units, and each up-sampling unit can be composed of a convolution layer and an activation function.

[0049] To sum up, according to the low-light video enhancement method of the present disclosure, the low-light video frame to be enhanced and the adjacent frame of the low-light video frame to be enhanced can be obtained; the low-light video frame to be enhanced and the adjacent frame are input into the trained low-light video enhancement model; the first video frame feature and the target difference image feature of the low-light video frame to be enhanced and the adjacent frame are determined by using the low-light video enhancement model; the target video frame feature is determined according to the first video frame feature and the target difference image feature, and the target video frame feature is subjected to low-light enhancement processing to obtain the low-light enhanced target video frame. For the embodiment of the present disclosure, the difference image can provide more high-frequency information for low-light enhancement, and by combining the first video frame feature and the target difference image feature, details can be more effectively restored, the definition of the video can be improved, and the blurring problem caused by noise can be reduced.

[0050] Further, as a refinement and extension of the above-mentioned embodiments, in order to completely describe the specific implementation process of the method of the present embodiment, the present embodiment provides a specific method as shown in Figure 2 The method comprises the following steps:

[0051] Step 201, performing feature alignment processing on the second video frame feature of the low-light video frame to be enhanced and the third video frame feature of the adjacent frame by using the low-light video enhancement model.

[0052] The second video frame feature can be an initial feature representation extracted from the low-light video frame to be enhanced, and the second video frame feature can include brightness, contrast, color, texture and other features of the low-light video frame to be enhanced;

[0053] The third video frame feature can be an initial feature representation extracted from the adjacent frame, and the third video frame feature can include brightness, contrast, color, texture and other features of the adjacent frame.

[0054] For the embodiment of the present disclosure, the second video frame feature of the low-light video frame to be enhanced and the third video frame feature of the adjacent frame are subjected to feature alignment processing by using the low-light video enhancement model, and as a possible implementation manner, the low-light video frame to be enhanced and the adjacent frame of the low-light video frame to be enhanced can be input into the multi-frame feature alignment fusion module of the low-light video enhancement model.

[0055] The multi-frame feature alignment fusion module can be composed of a modulated deformable convolution network and a spatial attention-based Transformer fusion module;

[0056] The deformable convolution can be used to adapt to the shape and movement of the object, so that a special convolution operation that better captures the feature change between frames; the modulated deformable convolution network can include four bias convolution layers and deformable convolution layer processing, the bias convolution layer can be used to calculate the offset value of each weight, and the deformable convolution layer can be used to perform deformable convolution according to the bias value, to output the continuous video frame feature maps (i.e., the second video frame feature and the third video frame feature) aligned at the feature level. This process can be performed four times, each time updating the feature map to capture more detailed information.

[0057] As a possible implementation, the feature-extracted continuous video frames (i.e., the dark-light video frame to be enhanced and its adjacent frames) can be input into the modulated deformable convolution network, the bias convolution layer of the modulated deformable convolution network is used to perform bias convolution calculation on the second video frame feature of the dark-light video frame to be enhanced and the third video frame feature of the adjacent frame, to obtain the offset value corresponding to the second video frame feature and the third video frame feature respectively; the deformable convolution layer of the modulated deformable convolution network is used to perform deformable convolution processing according to the bias value, to align the second video frame feature and the third video frame feature.

[0058] In step 202, the second video frame feature and the third video frame feature after the feature alignment processing are subjected to feature fusion processing based on a spatial attention mechanism, to obtain the fused first video frame feature.

[0059] The Transformer fusion module included in the multi-frame feature alignment and fusion module can be a module applying a spatial attention mechanism, and the Transformer fusion module can be used to fuse the features of the second video frame feature and the third video frame feature, thereby enhancing the feature representation of the current video frame; the spatial attention mechanism can be used to help the dark-light video enhancement model focus on the most important features in the current video frame and ignore the unimportant parts.

[0060] For the embodiments of the present disclosure, the aligned continuous video frame feature maps (i.e., the second video frame feature and the third video frame feature) can be input into the spatial attention-based Transformer fusion module, the spatial attention-based Transformer module can include three branches (i.e., a first branch, a second branch and a third branch) of Query, Key and Value, and each branch can include a normalization layer and a QKV convolution layer. After the convolution and normalization processing, the Q branch and the K branch can perform matrix multiplication to obtain a similarity matrix between spatial patches. The similarity matrix can be multiplied with the output result of the V branch to obtain an enhanced feature.

[0061] Finally, the enhanced feature can be connected with the original feature (i.e., the second video frame feature) in residual connection, and the feature after the residual connection is input into a Multilayer Perceptron (MLP) network to obtain the fused first video frame feature.

[0062] Among them, the Q branch can be used to input the current video frame feature (i.e., the second video frame feature), and the K and V branches can be used to input the adjacent video frame feature (i.e., the third video frame feature). The second video frame feature and the third video frame feature can be divided into 8x8 patches in the spatial dimension for more detailed feature processing.

[0063] Correspondingly, the second video frame feature and the third video frame feature after the feature alignment processing based on the spatial attention mechanism are subjected to feature fusion processing to obtain the fused first video frame feature, which can specifically include:

[0064] The first branch, the second branch and the third branch based on the spatial attention mechanism are used to process the second video frame feature and the third video frame feature to obtain the enhanced feature.

[0065] The enhanced feature and the second video frame feature are connected in residual connection to obtain the fused first video frame feature.

[0066] Step 203, using the low-light video enhancement model to calculate the difference image of the to-be-enhanced low-light video frame and the adjacent frame to obtain the target difference image feature.

[0067] For the embodiments of the present disclosure, the to-be-enhanced low-light video frame and its adjacent frame can be input into the difference image feature processing module of the low-light video enhancement model, the difference image of the to-be-enhanced low-light video frame is calculated using the to-be-enhanced low-light video frame and its adjacent frame, the calculated difference image is input into the feature extraction layer, and the difference image feature is extracted from the difference image using the feature extraction layer. The difference image feature can be mapped to a low-dimensional latent space using a VQGAN-based feature denoising module, and the interference noise in the difference image can be removed using a codebook discretization method.

[0068] For the embodiments of the present disclosure, as a possible implementation method, the difference image of the to-be-enhanced low-light video frame is calculated using the to-be-enhanced low-light video frame and its adjacent frame, and the difference image of the to-be-enhanced low-light video frame is calculated using the difference image calculation formula as follows:

[0069] D(x,y)=I n (x,y)-I n±1 (x,y)|

[0070] In the formula, D(x,y) can represent the pixel value of the difference image at the x, y position (i.e., the preset pixel point), In (x, y) can represent the pixel value of the nth frame of video frame (i.e. the dark light video frame to be enhanced) at the x, y position (i.e. the preset pixel point), I n±1 (x, y) can represent the pixel value of the nth ± 1 frame of video frame (i.e. the adjacent frame) at the x, y position (i.e. the preset pixel point), || can represent the absolute value.

[0071] The difference image obtained by the above formula can be input to a difference image feature processing module, and the difference image feature processing module can include a VQGAN feature encoding and discretization module.

[0072] The VQGAN encoding module can be used to convert the difference image from a high-dimensional feature to a low-dimensional space representation;

[0073] The discretization module can use the codebook method to perform discretization operation on the low-dimensional space feature. The codebook can be a pre-defined set, which can include a set of vectors representing low-dimensional space features. By comparing with the vectors in the codebook, each low-dimensional feature vector can find the closest codebook vector, thereby realizing the discretization of the feature.

[0074] In this way, while reducing the occupation of computing resources, the noise interference in the difference image feature is also reduced.

[0075] Correspondingly, the dark light video enhancement model is used to calculate the difference image of the dark light video frame to be enhanced and the adjacent frame to obtain the target difference image feature, which can specifically include:

[0076] Extracting the first pixel value of the preset pixel point of the dark light video frame to be enhanced, and extracting the second pixel value of the preset pixel point of the adjacent frame;

[0077] The absolute value of the pixel difference value of the first pixel value and the second pixel value is determined as the target pixel value of the preset pixel point of the dark light video frame to be enhanced and the adjacent frame;

[0078] According to the target pixel value, the difference image of the dark light video frame to be enhanced and the adjacent frame is determined;

[0079] Extracting the difference image feature in the difference image;

[0080] Discretizing the difference image feature to obtain the target difference image feature.

[0081] Step 204, based on the channel attention mechanism, the first video frame feature and the target difference image feature are fused to obtain the fused target video frame feature.

[0082] For the embodiments of the present disclosure, the processed first video frame features and the processed difference image features can be simultaneously input to three layers of down-sampling layers (i.e., down-sampling units) of a U-net network. The U-net can be a convolutional neural network including an encoder (down-sampling layer) and a decoder (up-sampling layer).

[0083] In the down-sampling module, the down-sampling module can include down-sampling units and a fusion unit. Each down-sampling unit can be composed of a convolutional layer and an activation function. There can be three down-sampling units in total. The convolutional layer can be used to extract features, and the activation function (such as ReLU) can be used to increase nonlinearity.

[0084] The output results (i.e., difference image features and first video frame features) of the three down-sampling units can be input to the fusion unit for feature fusion processing to enhance the original features (which can be second video frame features). The fusion unit can be a Transformer module based on a channel attention mechanism. The channel attention mechanism can allow the network to focus on more important feature channels, thereby improving the representation of the features. The Transformer fusion module can use a self-attention mechanism to capture the mutual relationship between different features.

[0085] The Transformer module based on the channel attention mechanism can include three branches (i.e., a first branch, a second branch, and a third branch) of Query, Key, and Value.

[0086] Each branch can include a normalization layer and a QKV convolutional layer. After convolution and normalization processing, the Q branch and the K branch can perform matrix multiplication to obtain a similarity matrix between channels. The similarity matrix can be multiplied with the output results of the V branch to obtain enhanced features.

[0087] The Q branch can be used to input the first video frame features, and the K and V branches can be used to input the target difference image features.

[0088] Finally, the enhanced features can be connected in residual to the original features (i.e., second video frame features) to obtain fused target video frame features. The residual-connected target video frame features can be input to a dark-light enhancement module to obtain dark-light enhanced video frame features. The dark-light enhanced features can be input to the symmetric three layers of up-sampling layers (which can be Multilayer Perceptron (MLP) networks) of the U-net to obtain dark-light enhanced target video frames.

[0089] The up-sampling layers can be used to restore the size of the feature map to the size of the original video frame, thereby generating enhanced video frames.

[0090] Correspondingly, the first video frame feature and the target difference image feature are subjected to feature fusion processing based on the channel attention mechanism to obtain a fused target video frame feature, which can specifically include:

[0091] The first branch, the second branch and the third branch based on the channel attention mechanism are used to perform feature processing on the first video frame feature and the target difference image feature to obtain an enhanced feature;

[0092] The enhanced feature and the second video frame feature are subjected to residual connection to obtain the fused target video frame feature.

[0093] For the embodiments of the present disclosure, the main framework of the dark-light video enhancement model can be a double-branch U-net network, which can include two main branches of an original video frame processing branch and a difference image processing branch. The original video frame processing branch can be used to process the video frame features of the original dark-light video frame to be enhanced and the adjacent frame of the dark-light video frame to be enhanced. The difference image processing branch can be used to process the difference image of the input dark-light video frame to be enhanced and the adjacent frame of the dark-light video frame to be enhanced.

[0094] In the dark-light video enhancement model, the original video frame processing branch and the difference image processing branch can perform feature fusion when down-sampling, that is, the features extracted by the two branches are combined to create a more rich feature representation. The fused features can be processed by a bottleneck module and an up-sampling layer to generate a video frame under normal lighting conditions.

[0095] The bottleneck module can be used to reduce the size of the feature map, and the up-sampling layer can be used to restore the feature map to the size of the original video frame, while generating an enhanced video frame.

[0096] The training process of the dark-light video enhancement model can specifically include:

[0097] Initialize the dark-light enhancement model based on the difference image, and initialize the model parameters of all modules (such as the multi-frame feature alignment and fusion module, the difference image feature processing module, the U-net, the dark-light enhancement module, etc.). In this embodiment, the initial dark-light enhancement model can be constructed based on a deep learning framework such as Pytorch.

[0098] Obtain training samples, which can use DID and SDSD datasets as training samples, which contain paired dark-light videos (which can include dark-light video frames, adjacent frames of dark-light video frames) and normal light videos (which can include normal light video frames), and the dark-light videos and the normal light videos are aligned frame by frame;

[0099] The dark-light video frame and the adjacent frame can be taken as input features of the dark-light video enhancement model, the normal-light video frame can be taken as a training label of the dark-light video enhancement model, and model parameters of the dark-light video enhancement model are continuously adjusted until the iteration number of the dark-light video enhancement model is greater than or equal to a preset iteration number, or a loss function value of the dark-light video enhancement model is less than a preset loss function threshold, and it is determined that the dark-light video enhancement model is trained, wherein the loss function includes a reconstruction loss function, a perception loss function and a structure loss function.

[0100] Specifically, the original image (i.e., the dark-light video frame and the adjacent frame of the dark-light video frame) in the training sample can be input to the multi-frame feature alignment fusion module for alignment processing, and the aligned adjacent frame feature is used to perform feature enhancement on the current video frame to obtain enhanced video frame features.

[0101] The original image (i.e., the dark-light video frame and the adjacent frame of the dark-light video frame) in the training sample can be input to the differential image feature processing module to obtain corresponding denoised differential image features.

[0102] The enhanced video frame features and the differential image features can be input to the down-sampling layer of the U-net, and the differential image features and the video frame features are fused at each down-sampling layer to obtain video frame features enhanced by the differential image features.

[0103] The video frame features enhanced by the differential image features can be input to the dark-light enhancement module for dark-light enhancement processing to obtain dark-light enhanced video frame features, and the dark-light enhanced video frame features can be input to the up-sampling layer to obtain dark-light enhanced video frames.

[0104] The model parameters of the multi-frame feature alignment fusion module, the differential image feature processing module, the down-sampling fusion module, the dark-light enhancement module and the up-sampling module can be adjusted according to the difference between the enhanced video output by the dark-light video enhancement model and the corresponding real normal-light video, and when the iteration number reaches the preset iteration number, the training is terminated to obtain the trained dark-light video enhancement model based on the differential image features.

[0105] For the embodiments of the present disclosure, the specific module execution process is as shown in Figure 3 The video acquisition module can be used to acquire the dark-light video to be enhanced.

[0106] The dark-light video enhancement module can be used to input the dark-light video to be enhanced to obtain enhanced video frames. The dark-light video enhancement module can include a multi-frame feature alignment fusion module, a differential image feature processing module, a down-sampling fusion module, a dark-light enhancement module and an up-sampling module.

[0107] The multi-frame feature alignment fusion module can be used to align the features of the adjacent frames and fuse the aligned features to the original features.

[0108] The differential image feature processing module can be used to obtain a differential image feature removed from noise interference.

[0109] The down-sampling fusion module can be used to down-sample the video frame feature and the differential image feature, and fuse the video frame feature and the differential image feature.

[0110] The dark light enhancement module can be used to obtain a dark light enhanced video frame feature.

[0111] The up-sampling module can be used to obtain a dark light enhanced video frame.

[0112] For the embodiments of the present disclosure, after obtaining the enhanced video output by the dark light video enhancement model, the model can be trained using a loss function.

[0113] The loss function of the dark light video enhancement model can be composed of a reconstruction loss function, a perception loss function and a structure loss function. The reconstruction loss can be a L1 loss weighted according to brightness, with a weight of 4 for the darkest 40% region and a weight of 1 for other regions. The perception loss can be a difference value based on feature extraction by a VGG network. The structure loss can be composed of an SSIM loss and an MS-SSIM loss.

[0114] The SSIM loss function is as follows:

[0115]

[0116] In the formula, μ x and μ y may be the average value of pixel values, and may be variances, σ xy may be covariances, and C1 and C2 can be constants to prevent the denominator from being zero.

[0117] The perception loss formula is as follows:

[0118]

[0119] In the formula, E and G can be an enhanced image and a real image, W i,j , H i,j and C i, can be used to describe the dimensions of respective feature maps in the VGG network, φ i, may represent a feature map obtained by the i-th block and the j-th convolutional layer in the VGG-19 network.

[0120] The weighted reconstruction loss formula is as follows:

[0121]

[0122] In the formula, EL and G L E can enhance low-light areas of both real and enhanced images. H and G H This can be the rest of the image. It can take the value w. L 4, w H The value is 1.

[0123] For embodiments of this disclosure, such as Figure 4 As shown, the input to the low-light video enhancement model is the current frame and its neighboring frames. The neighboring frames can contain only the previous N frames, the next N frames, or both. The difference image is obtained based on the input and the difference image calculation formula. The multi-frame input sequentially passes through a feature extraction module, a DCNv2 multi-frame feature alignment module, and a multi-frame fusion module based on spatial attention Transformer to obtain the original features after feature enhancement. The difference image sequentially passes through a difference feature extraction module, a VQGAN encoding module, and a VQGAN discretization module to obtain the difference features after feature enhancement. Then, the original features and the difference features are input into a three-layer downsampling fusion module. The fusion module consists of a structure where each layer inputs the original and difference features into a convolutional layer and a ReLU activation function for downsampling. The downsampled original and difference features are then input into a feature fusion module based on channel attention (Transformer). The output of the fusion module is used as the input to the original features of the next layer, and the downsampled difference features are used as the input to the difference features of the next layer. The original features output from the downsampling fusion module are then input into a residual module for dark enhancement. The output of the residual module is then input into a three-layer upsampling module, each containing a convolutional layer and a ReLU activation function. The upsampling module outputs the final result.

[0124] In summary, according to the low-light video enhancement method of this disclosure, the low-light video frame to be enhanced and its neighboring frames can be obtained; the low-light video frame to be enhanced and its neighboring frames are input into a trained low-light video enhancement model; the low-light video enhancement model is used to determine the first video frame features and target differential image features of the low-light video frame to be enhanced and its neighboring frames; the target video frame features are determined based on the first video frame features and the target differential image features, and low-light enhancement processing is performed on the target video frame features to obtain the low-light enhanced target video frame. In the embodiments of this disclosure, the differential image can provide more high-frequency information for low-light enhancement. By combining the first video frame features and the target differential image features, details can be recovered more effectively, video clarity can be improved, and blurring caused by noise can be reduced.

[0125] Based on the above Figure 1 , Figure 2 The specific implementation of the method shown in this embodiment provides a low-light video enhancement device, such as... Figure 5As shown, the apparatus comprises: an acquisition module 31, an input module 32, a determination module 33, a processing module 34;

[0126] The acquisition module 31 is configured to acquire a to-be-enhanced dark-light video frame and a neighboring frame of the to-be-enhanced dark-light video frame.

[0127] The input module 32 is configured to input the to-be-enhanced dark-light video frame and the neighboring frame into a trained dark-light video enhancement model.

[0128] The determination module 33 is configured to determine, by using the dark-light video enhancement model, first video frame features of the to-be-enhanced dark-light video frame and the neighboring frame and target difference image features.

[0129] The processing module 34 is configured to determine target video frame features according to the first video frame features and the target difference image features, and perform dark-light enhancement processing on the target video frame features to obtain dark-light enhanced target video frames.

[0130] In a specific application scenario, the determination module 33 can be configured to perform feature alignment processing on second video frame features of the to-be-enhanced dark-light video frame and third video frame features of the neighboring frame by using the dark-light video enhancement model.

[0131] The second video frame features and the third video frame features after the feature alignment processing are subjected to feature fusion processing based on a spatial attention mechanism to obtain the first video frame features after the fusion.

[0132] In a specific application scenario, the determination module 33 can be configured to perform difference image calculation on the to-be-enhanced dark-light video frame and the neighboring frame by using the dark-light video enhancement model to obtain the target difference image features.

[0133] In a specific application scenario, the processing module 34 can be configured to perform feature fusion processing on the first video frame features and the target difference image features based on a channel attention mechanism to obtain the target video frame features after the fusion.

[0134] In a specific application scenario, the determination module 33 can be configured to perform bias convolution calculation on the second video frame features and the third video frame features to obtain offset values corresponding to the second video frame features and the third video frame features, respectively.

[0135] Deformation convolution processing is performed based on the offset values to align the second video frame features and the third video frame features.

[0136] In a specific application scenario, the determining module 33 can be configured to perform feature processing on the second video frame feature and the third video frame feature based on the first branch, the second branch and the third branch of the spatial attention mechanism, to obtain an enhanced feature.

[0137] The enhanced feature and the second video frame feature are connected in residual, to obtain the first video frame feature after fusion.

[0138] In a specific application scenario, the determining module 33 can be configured to extract a first pixel value of the preset pixel point of the to-be-enhanced dark-light video frame, and extract a second pixel value of the preset pixel point of the adjacent frame;

[0139] The absolute value of the pixel difference value of the first pixel value and the second pixel value is determined as a target pixel value of the preset pixel point of the to-be-enhanced dark-light video frame and the adjacent frame;

[0140] According to the target pixel value, a difference image of the to-be-enhanced dark-light video frame and the adjacent frame is determined;

[0141] The difference image feature in the difference image is extracted;

[0142] The difference image feature is discretized to obtain the target difference image feature.

[0143] In a specific application scenario, the processing module 34 can be configured to perform feature processing on the first video frame feature and the target difference image feature based on the first branch, the second branch and the third branch of the channel attention mechanism, to obtain an enhanced feature;

[0144] The enhanced feature and the second video frame feature are connected in residual, to obtain the target video frame feature after fusion.

[0145] In a specific application scenario, as shown in Figure 5 The device further includes a training module 35.

[0146] The training module 35 is configured to obtain a training sample, the training sample including a dark-light video frame, an adjacent frame of the dark-light video frame, and a normal-light video frame corresponding to the dark-light video frame and the adjacent frame.

[0147] The dark-light video frame and the adjacent frame are taken as input features of the dark-light video enhancement model, the normal-light video frame is taken as a training label of the dark-light video enhancement model, and model parameters of the dark-light video enhancement model are continuously adjusted until an iteration number of the dark-light video enhancement model is greater than or equal to a preset iteration number, or a loss function value of the dark-light video enhancement model is less than a preset loss function threshold, and it is determined that the dark-light video enhancement model is trained to be completed, wherein the loss function includes a reconstruction loss function, a perception loss function and a structure loss function.

[0148] It should be noted that other corresponding descriptions of the functions of the dark-light video enhancement device provided in the embodiment are described in the corresponding descriptions of the methods in Figure 1 and Figure 2 , and will not be described here.

[0149] Based on the methods shown in Figure 1 and Figure 2 , accordingly, the present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods shown in Figure 1 and Figure 2 .

[0150] Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of various implementation scenarios of the present disclosure.

[0151] Based on the methods shown in Figure 1 and Figure 2 , and Figure 5 the virtual device embodiment, in order to achieve the above purpose, the present disclosure embodiment also provides an electronic device, such as a smart phone, a tablet computer, a drone, a smart robot, etc. The smart terminal includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the methods shown in Figure 1 and Figure 2 .

[0152] Optionally, the above-mentioned entity device can further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, and the like. The user interface can include a display screen, an input unit such as a keyboard, and the like. Optionally, the user interface can further include a USB interface, a card reader interface, and the like. The network interface can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and the like.

[0153] Those skilled in the art can understand that the above-mentioned entity device structure provided by the present disclosure does not constitute a limitation on the entity device, and can include more or fewer components, or combine certain components, or different component arrangements.

[0154] The storage medium can also include an operating system and a network communication module. The operating system is a program that manages hardware and software resources of the above-mentioned entity device, and supports the running of information processing programs and other software and / or programs. The network communication module is used to realize communication between components inside the storage medium, and communication with other hardware and software in the information processing entity device.

[0155] Based on the above-mentioned methods as shown in Figure 1 and Figure 2 , and the virtual device embodiments as shown in Figure 5 , the present embodiment also provides a chip including one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from the memory of an electronic device and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the above-mentioned methods as shown in Figure 1 and Figure 2 .

[0156] Those skilled in the art can clearly understand from the description of the above embodiments that the present disclosure can be implemented by means of software with a necessary general hardware platform, or by hardware. The dark-light video enhancement method, device and electronic equipment provided by the present disclosure, compared with the prior art, acquire a dark-light video frame to be enhanced and a neighboring frame of the dark-light video frame to be enhanced; input the dark-light video frame to be enhanced and the neighboring frame into a dark-light video enhancement model trained; determine first video frame features and target difference image features of the dark-light video frame to be enhanced and the neighboring frame by using the dark-light video enhancement model; determine target video frame features according to the first video frame features and the target difference image features, and perform dark-light enhancement processing on the target video frame features to obtain a dark-light enhanced target video frame. For the embodiments of the present disclosure, the difference image can provide more high-frequency information for dark-light enhancement, and by combining the first video frame features and the target difference image features, details can be more effectively restored, the clarity of the video can be improved, and the blurring problem caused by noise can be reduced.

[0157] It should be noted that, in this document, the terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0158] The above is only a specific embodiment of the present application, which enables those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. A method for enhancing low-light video, characterized in that, include: Obtain the low-light video frame to be enhanced and its adjacent frames; The low-light video frame to be enhanced and the adjacent frames are input into the trained low-light video enhancement model; The low-light video enhancement model is used to determine the first video frame features and target differential image features of the low-light video frame to be enhanced and the adjacent frames; The target video frame features are determined based on the first video frame features and the target differential image features, and the target video frame features are subjected to dark light enhancement processing to obtain the target video frame after dark light enhancement.

2. The method according to claim 1, characterized in that, The step of determining the first video frame features of the low-light video frame to be enhanced and the adjacent frames using the low-light video enhancement model includes: The low-light video enhancement model is used to perform feature alignment processing on the second video frame features of the low-light video frame to be enhanced and the third video frame features of the adjacent frames; Based on the spatial attention mechanism, feature fusion processing is performed on the feature-aligned features of the second video frame and the features of the third video frame to obtain the fused features of the first video frame.

3. The method according to claim 1, characterized in that, The step of determining the target difference image features of the low-light video frame to be enhanced and the adjacent frames using the low-light video enhancement model includes: The low-light video enhancement model is used to perform differential image calculation on the low-light video frame to be enhanced and the adjacent frames to obtain the target differential image features.

4. The method according to claim 1, characterized in that, Determining the target video frame features based on the first video frame features and the target differential image features includes: The first video frame features and the target differential image features are fused based on the channel attention mechanism to obtain the fused target video frame features.

5. The method according to claim 2, characterized in that, The step of using the low-light video enhancement model to perform feature alignment processing on the second video frame features of the low-light video frame to be enhanced and the third video frame features of the adjacent frames includes: The second video frame feature and the third video frame feature are subjected to bias convolution calculation to obtain the offset values ​​corresponding to the second video frame feature and the third video frame feature respectively; Deformed convolution processing is performed based on the offset value to align the features of the second video frame and the features of the third video frame.

6. The method according to claim 2, characterized in that, The feature fusion process, which involves merging the second video frame features and the third video frame features after feature alignment based on a spatial attention mechanism, to obtain the fused first video frame features, includes: Based on the first, second, and third branches of the spatial attention mechanism, feature processing is performed on the features of the second video frame and the features of the third video frame to obtain enhanced features; The enhanced features and the second video frame features are residually concatenated to obtain the fused first video frame features.

7. The method according to claim 3, characterized in that, The step of using the low-light video enhancement model to perform differential image calculation on the low-light video frame to be enhanced and the adjacent frames to obtain the target differential image features includes: Extract the first pixel value of the low-light video frame to be enhanced at a preset pixel point, and extract the second pixel value of the adjacent frame at the preset pixel point; The absolute value of the pixel difference between the first pixel value and the second pixel value is determined as the target pixel value of the low-light video frame to be enhanced and the adjacent frame at the preset pixel point; The difference image between the low-light video frame to be enhanced and the adjacent frame is determined based on the target pixel value; Extract the difference image features from the difference image; The differential image features are discretized to obtain the target differential image features.

8. The method according to claim 4, characterized in that, The feature fusion process based on the channel attention mechanism for the first video frame features and the target differential image features to obtain the fused target video frame features includes: Based on the first, second, and third branches of the channel attention mechanism, feature processing is performed on the features of the first video frame and the features of the target difference image to obtain enhanced features; The enhanced features and the second video frame features are residually concatenated to obtain the fused target video frame features.

9. The method according to claim 1, characterized in that, The training process of the low-light video enhancement model includes: Acquire training samples, which include dark light video frames, adjacent frames of the dark light video frames, and normal light video frames corresponding to the dark light video frames and the adjacent frames. The low-light video frames and their adjacent frames are used as input features of the low-light video enhancement model, and the normal-light video frames are used as training labels for the low-light video enhancement model. The model parameters of the low-light video enhancement model are continuously adjusted until the number of iterations of the low-light video enhancement model is greater than or equal to a preset number of iterations, or the loss function value of the low-light video enhancement model is less than a preset loss function threshold. At this point, the training of the low-light video enhancement model is determined to be complete. The loss function includes a reconstruction loss function, a perceptual loss function, and a structural loss function.

10. A low-light video enhancement device, characterized in that, include: The acquisition module is used to acquire the low-light video frame to be enhanced and its adjacent frames; The input module is used to input the low-light video frame to be enhanced and the adjacent frames into the trained low-light video enhancement model; The determination module is used to determine the first video frame features and target differential image features of the low-light video frame to be enhanced and the adjacent frames using the low-light video enhancement model; The processing module is used to determine the target video frame features based on the first video frame features and the target differential image features, and to perform dark light enhancement processing on the target video frame features to obtain the dark light enhanced target video frame.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 9.

12. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 9.

13. A chip, characterized in that, The device includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of an electronic device and send the signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the method of any one of claims 1 to 9.

Images