Low-light video processing methods, equipment and storage media

Abstract

The present application discloses a low-light video processing method, device and storage medium, which belongs to the field of video image processing technology. The method includes: obtaining the same number of previous frame images and subsequent frame images of the current video frame of the low-light video to obtain a frame image set of the current video frame; and traversing the low-light video to obtain a frame image set of all video frames. After all frame images in the frame image set of all video frames are aligned, they are input into a pre-trained low-light enhancement model to obtain enhanced frame images. Based on the enhanced frame images, an enhanced video is generated. The technical solution of the present application can achieve low-light video quality enhancement.

Description

Technical Field

[0001] This application relates to the field of video image processing technology, and in particular to a low-light video processing method, device and storage medium. Background Technology

[0002] Numerous studies have shown that video information occupies a significant proportion of modern information flow. The ambient light intensity during video recording has a substantial impact on video quality. Low-light video refers to video shot under conditions of low ambient light intensity. This type of low-light video typically suffers from insufficient light entering the shooting device due to low ambient light levels, resulting in problems such as low illumination, low contrast, unclear details, and high noise. Therefore, there is an urgent need for a low-light video processing method to enhance the image quality of low-light images. Summary of the Invention

[0003] The main objective of this application is to provide a low-light video processing method, device, and storage medium, which aims to enhance the quality of low-light images.

[0004] To achieve the above objectives, this application provides a low-light video processing method, which includes the following steps: acquiring the same number of previous and subsequent frame images as the current video frame of a low-light video to obtain a set of frame images for the current video frame; and traversing the low-light video to obtain a set of frame images for all video frames. After image alignment of all frame images in the set of frame images for all video frames, the images are input into a pre-trained low-light enhancement model to obtain enhanced frame images. Based on the enhanced frame images, an enhanced video is generated.

[0005] To achieve the above objectives, embodiments of this application provide a low-light video processing device. The device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. When the program is executed by the processor, it implements the steps of the low-light video processing method described in any one of these applications.

[0006] To achieve the above objectives, embodiments of this application provide a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the low-light video processing method described in any one of these applications.

[0007] The low-light video processing method, device, and storage medium provided in this application obtain a set of frame images for the current video frame by acquiring the same number of previous and subsequent frame images for the current video frame of a low-light video, and then traversing the low-light video to obtain a set of frame images for all video frames of the low-light video. After image alignment of the set of frame images for all video frames, the images are input into a pre-trained low-light enhancement model to obtain enhanced frame images. Based on the enhanced frame images, an enhanced video is generated, achieving low-light video image quality enhancement and improving the brightness, color, contrast, and detail of the video. Furthermore, by using a set of frame images including multiple frames after image alignment as input to the low-light enhancement model, compared to low-light enhancement models in related technologies that use a single frame image as input, this low-light enhancement model improves the richness of the acquired video frame information while ensuring the accuracy of the acquired video frame information due to the high correlation and similarity of the input multiple frame images, thereby improving the enhancement effect of the low-light enhancement model on video frames. Attached Figure Description

[0008] Figure 1 This is a flowchart of a low-light video processing method provided in an embodiment of this application.

[0009] Figure 2 This is a flowchart of a frame image set alignment provided in an embodiment of this application.

[0010] Figure 3 This is a schematic diagram illustrating the principle of a motion estimation model provided in an embodiment of this application.

[0011] Figure 4 This is a schematic diagram illustrating the principle of a low-light enhancement model provided in an embodiment of this application.

[0012] Figure 5 This application provides a flowchart of a method for obtaining enhanced frame images based on a low-light enhancement model.

[0013] Figure 6 This is a partial schematic diagram of the principle of a low-light enhancement model provided in the embodiments of this application.

[0014] Figure 7 This is a schematic diagram of a process for determining an enhanced frame image provided in an embodiment of this application.

[0015] Figure 8 This is a schematic diagram of the structure of a low-light video processing device provided in an embodiment of this application.

[0016] Figure 9 This is a schematic diagram of another low-light video processing device provided in an embodiment of this application. Detailed Implementation

[0017] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0018] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustration and has no inherent meaning. Therefore, "module," "part," or "unit" may be used interchangeably.

[0019] Please refer to Figure 1 This document illustrates a flowchart of a low-light video processing method provided in this embodiment. This method can be applied to a low-light video processing device, which can be an electronic device, such as a mobile terminal, personal computer, or wearable device. Figure 1 As shown, the method may include:

[0020] Step S110: Obtain the same number of previous and next frame images of the current video frame of the low-light video to obtain the frame image set of the current video frame; and traverse the low-light video to obtain the frame image set of all video frames.

[0021] Low-light video consists of one or more frames (video frames). The preceding frame of the current video frame refers to the frame image preceding the current video frame in the playback sequence of the low-light video. The following frame of the current video frame refers to the frame image following the current video frame in the playback sequence of the low-light video. The preceding and following frames can be referred to as adjacent frames of the current video frame. The set of frame images for any video frame (e.g., the current video frame) includes that video frame, n preceding frames, and n following frames, where n is a positive integer and n ≥ 1. For example, 3 ≥ n ≥ 1. For instance, if n = 1, the set of frame images for any video frame includes that video frame, the preceding frame, and the following frame. Accordingly, acquiring the same number of preceding and following frames for the current video frame of the low-light video includes acquiring one preceding frame and one following frame for the current video frame of the low-light video.

[0022] In one embodiment, the low-light video can be a low-light video with the same or similar scene content (also known as motion scene content). That is, the scene content of each video frame in the low-light video should be the same or similar. For example, the background of each video frame in the low-light video is the same. For example, the background of each video frame in the low-light video is a forest, a sea, or the same room. In this way, since the content of each video frame in the low-light video is the same or similar, the similarity between the previous frame image and the following frame image of the current video frame in the low-light video is high. When the frame image set composed of highly similar multi-frame images is input into the subsequent low-light enhancement model, it can ensure that the model obtains more input features due to the simultaneous input of multiple frames, while avoiding the large interference of features from multiple frames with large differences on the features of the current video frame, thereby improving the accuracy of the input features obtained by the model.

[0023] In one embodiment, if the number of preceding and following frames of the current video frame is different, the current video frame can be copied as either the preceding or following frame until the number is the same. The low-light video processing device can preset a number (n) of preceding and following frames to be acquired for the current video frame. It is then determined whether the number of acquiring preceding and following frames corresponding to the current video frame is the same as the preset number. If the numbers are the same, the low-light video processing device acquires a preset number of preceding and following frame images for the current video frame, obtaining a set of frame images corresponding to the current video frame. If they are different, and the number of preceding frame images is less than the number of following frame images, the low-light video processing device can copy the current video frame multiple times until the sum of the number of copied current video frames and the number of preceding frame images equals the number of following frame images. The copied current video frame is then used as the preceding frame image, obtaining a set of frame images corresponding to the current video frame. Similarly, if the numbers are different, and the number of following frame images is less than the preceding frame images, the low-light video processing device can copy the current video frame multiple times until the sum of the number of copied current video frames and the number of following frame images equals the number of preceding frame images. The copied current video frame is then used as the following frame image, obtaining a set of frame images corresponding to the current video frame. Furthermore, the low-light video is traversed to obtain a set of frame images for each video frame in all the video frames included in the low-light video.

[0024] In another embodiment, when 3 ≥ n ≥ 1, since low-light videos typically consist of more than 3 frames, usually only the number of frames preceding the first frame and the number of frames following the last frame differ from the preset number n. Therefore, for a low-light video segment, the low-light video processing device can directly copy the current video frame n times as preceding frames when it determines that the current video frame is the first frame of the low-light video, thus obtaining a set of frame images for the current video frame. Similarly, the low-light video processing device can directly copy the current video frame n times as following frames when it determines that the current video frame is the last frame of the low-light video, thus obtaining a set of frame images for the current video frame. For example, the low-light video processing device uses a first frame identifier to identify the first frame of the low-light video and a last frame identifier to identify the last frame of the low-light video. If a low-light video processing device acquires a current video frame with a first frame identifier, it can directly copy the current video frame n times as the previous frame image to obtain the frame image set of the current video frame. Similarly, if the low-light video processing device acquires a current video frame with a last frame identifier, it can directly copy the current video frame n times as the next frame image to obtain the frame image set of the current video frame. This avoids the need to compare the available previous and next frame images with a preset number, simplifying the process of obtaining the frame image set and reducing equipment overhead.

[0025] It should be noted that when any of the methods described herein are executed on low-light videos with the same or similar scene content (also known as motion scene content) in this embodiment, for low-light videos including different scene content, the low-light video processing device can set a scene switching trigger signal so that when the low-light video processing device obtains the scene switching trigger signal, it determines that the scene content in the low-light video has been switched, and then uses the low-light video segment after the scene content switch as a new low-light video, and executes any of the methods described in this embodiment on the new low-light video.

[0026] Step S120: Align all frame images in the set of all video frames and input them into the pre-trained low-light enhancement model to obtain the enhanced frame images.

[0027] The low-light video processing device performs image alignment on all frame images in the frame image set of each video frame, resulting in an image-aligned frame image set for each video frame. This image-aligned frame image set is then input into a pre-trained low-light enhancement model to obtain the enhanced frame image for each video frame.

[0028] In one embodiment, please refer to Figure 2 This illustrates a flowchart of frame image set alignment provided in this embodiment. For example... Figure 2 As shown, the process by which a low-light video processing device aligns all frame images in a set of all video frame images may include:

[0029] Step S210: Using the target video frame as a reference frame image in all video frames, perform motion estimation on each supplementary frame image in the frame image set of the target video frame, and sequentially determine the motion vector corresponding to each supplementary frame image in the frame image set of the target video frame. The supplementary frame image consists of the preceding frame image and the following frame image in the frame image set of the target video frame.

[0030] The target video frame is every video frame in the entire video frame set, including the current video frame. The motion vector corresponding to the supplementary frame image represents the motion deviation, or relative displacement, between each pixel in the supplementary frame image and each pixel in the target frame image. The low-light video processing device uses the target video frame as a reference frame image for motion estimation (ME), and sequentially determines the motion vector (MV) corresponding to each supplementary frame image in the target video frame set. There are several ways to implement this; this embodiment uses the following three implementation methods as examples.

[0031] In the first implementation, the low-light video processing device supplements each frame in the frame image set of the target video frame with a supplementary frame image, inputs the target video frame and the supplementary frame image into a pre-trained motion estimation model, and obtains the motion vector corresponding to the supplementary frame image, thereby obtaining the motion vector corresponding to each supplementary frame image in the frame image set of the target video frame.

[0032] The number of pre-trained motion estimation models is one less than the number of frames in the frame image set, i.e., there are 2^n motion estimation models, and each motion estimation model is identical. The low-light video processing device inputs the target video frame and each supplementary frame image in the frame image set of the target video frame into a pre-trained motion estimation model to obtain the motion vector corresponding to each supplementary frame image. This process is repeated to obtain the motion vector corresponding to each supplementary frame image in the frame image set of each video frame in the entire video frame.

[0033] For example, assuming n=1, the target video frame set contains 3 frames, including the target video frame l_lowt, the previous frame l_lowt-1, and the next frame l_lowt+1, where the previous frame l_lowt-1 and the next frame l_lowt+1 are supplementary frames. The number of motion estimation models is 2. The low-light video processing device inputs the target video frame l_lowt and the previous frame l_lowt-1 into one motion estimation model to obtain the motion vector corresponding to the previous frame l_lowt-1. The low-light video processing device inputs the target video frame l_lowt and the next frame l_lowt+1 into another motion estimation model to obtain the motion vector corresponding to the next frame l_lowt+1.

[0034] The motion estimation model is trained on a video frame in a low-light video and n frames before and n frames after the video frame, according to a first loss function. The input parameters of the first loss function include: the video frame, the n frames before the video frame, the n frames after the video frame, the first motion vector corresponding to the n frames before the video frame, and the second motion vector corresponding to the n frames after the video frame.

[0035] For example, the motion estimation model is a convolutional neural network (CNN) model. The training process of this motion estimation model may include: inputting a video frame from a low-light video segment, along with n preceding and n following frames of that video frame, into an initial CNN model to obtain a first motion vector corresponding to each of the n preceding frames and a second motion vector corresponding to each of the n following frames. Then, inputting the video frame, the n preceding and n following frames, the first motion vector, and the second motion vector into a first loss function to obtain a loss result. If the loss result is not within the convergence range of the function, adjusting the network parameters of the initial CNN model to obtain an adjusted CNN model. This input operation is repeated until the loss result of the first loss function reaches the convergence range of the function, thus confirming the obtained CNN model as the motion estimation model.

[0036] In one embodiment, taking n=1 as an example, the motion estimation model is obtained by training a video frame in a low-light video segment, along with one preceding frame and one following frame, using a first loss function. This first loss function is Loss... ME satisfy:

[0037] Loss ME =MSE(I_low t W b (I_low t-1 ,f t-1→t ))

[0038] +MSE(I_low t W b (I_low t+1 ,f t+1→t ));

[0039] l_low t Indicates a video frame. l_low t-1 This represents the image of the frame preceding the current video frame. l_low t+1 This represents the frame following the first frame of the video frame. t represents the time of the video frame. t-1 represents the time of the frame preceding the first frame of the video frame. t+1 represents the time of the frame following the first frame of the video frame. f t-1→t f represents the motion vector corresponding to the previous frame image determined by the motion estimation model. t+1→t This represents the motion vector corresponding to the previous frame image determined by the motion estimation model. MSE stands for Mean Squared Error (MSE). W b (I_low t-1 ,f t-1→t ) represents the previous frame image determined by applying motion compensation (e.g., affine transformation) to the previous frame image based on the motion vectors determined by the motion estimation model. b (I_low t+1 ,f t+1→t ) represents the motion vector corresponding to the subsequent frame image determined based on the motion estimation model, and the subsequent frame image is determined after motion compensation (e.g., image affine transformation) is applied to the subsequent frame image.

[0040] In this way, since the motion estimation model corresponding to the motion vector of each supplementary frame image is the same, the motion estimation model only needs to be trained once. It can also be understood that multiple motion estimates can share a set of network parameters, which reduces the resource consumption caused by training the motion estimation model.

[0041] In this embodiment, as shown Figure 3As shown, the process by which the motion estimation model determines the motion vector of a supplementary frame image can include: performing a first convolution process twice on the target video frame and the supplementary frame image using an activation function, conv&LeakyRelU 32Out. conv&LeakyRelU 32Out indicates that the LeakyRelU activation function is used for convolution processing, with 32 output channels. The data after the first convolution process is then subjected to a first pooling process. This first pooling process is MaxPooling 2X, where MaxPooling 2X represents reducing the width and height of the image by a factor of 2. The data after the first pooling process is then subjected to a second convolution process twice using the first activation function, conv&LeakyRelU 64Out. conv&LeakyRelU 64Out indicates that the LeakyRelU activation function is used for convolution processing, with 64 output channels. The data after the second convolution process is then subjected to a second pooling operation. The second pooling operation is MaxPooling 2X, which means reducing the image's width and height by a factor of 2. The data after the second pooling operation is then subjected to a third convolution process using the first activation function, conv&LeakyRelU 128Out. conv&LeakyRelU 128Out means using the first activation function LeakyRelU for convolution, resulting in 128 output channels. The data after the third convolution process is then subjected to a third pooling operation, MaxPooling 2X, which means reducing the image's width and height by a factor of 2. The data after the third pooling operation is then subjected to a fourth convolution process using the first activation function, conv&LeakyRelU 256Out. conv&LeakyRelU 256Out means using the first activation function LeakyRelU for convolution, resulting in 256 output channels. The data after the fourth convolution is upsampled for the first time. This first upsampling is called Bilinear 2X, which means that the width and height of the image are magnified by a factor of 2 using bilinear interpolation. The data after the first upsampling is then subjected to a fifth convolution twice using the first activation function: conv&LeakyRelU 128Out. conv&LeakyRelU 128Out means that the first activation function, LeakyRelU, is used for convolution, and the number of output channels is 128. The data after the fifth convolution is then upsampled a second time.The second upsampling process is Bilinear 2X, which means using bilinear interpolation to enlarge the image's width and height by a factor of 2. The data after the second upsampling process is then subjected to a sixth convolution process using the first activation function twice: conv&LeakyRelU 64Out. conv&LeakyRelU 64Out means using the first activation function LeakyRelU for convolution, with 64 output channels. The data after the sixth convolution process is then subjected to a third upsampling process: Bilinear 2X, which means using bilinear interpolation to enlarge the image's width and height by a factor of 2. The data after the third upsampling process is then subjected to a seventh convolution process using the first activation function: conv&LeakyRelU 32Out. conv&LeakyRelU 32Out means using the first activation function LeakyRelU for convolution, with 32 output channels. The data processed by the seventh convolution is then subjected to an eighth convolution using the second activation function, conv&Tanh 2Out. conv&Tanh 2Out indicates that the convolution is performed using the second activation function Tanh, with a total of 2 output channels.

[0042] In the second implementation, the low-light video processing device inputs the target video frame and the previous frame image into a pre-trained motion estimation model for the previous frame image to obtain the motion vector corresponding to the previous frame image. Similarly, it inputs the target video frame and the subsequent frame image into a pre-trained motion estimation model for the subsequent frame image to obtain the motion vector corresponding to the subsequent frame image.

[0043] The number of pre-trained motion estimation models is one less than the number of frames in the frame image set, i.e., there are 2^n motion estimation models, and each motion estimation model is different. These 2^n motion estimation models can be divided into two categories: one category is the motion estimation models for the previous frame image, with n models, each corresponding to one previous frame image; the other category is the motion estimation models for the subsequent frame image, with n models, each corresponding to one subsequent frame image.

[0044] The low-light video processing device takes a target video frame and a different preceding frame from its frame image set (used as supplementary frames) as inputs to a pre-trained motion estimation model corresponding to that preceding frame, obtaining the motion vector for that preceding frame. Similarly, it takes a target video frame and a different following frame from its frame image set (used as supplementary frames) as inputs to a pre-trained motion estimation model corresponding to that following frame, obtaining the motion vector for that following frame. This process is repeated to obtain the motion vectors for each preceding and following frame in the frame image set for each video frame across all video frames.

[0045] For example, assuming n=1, the target video frame set contains 3 frames, including the target video frame l_lowt, the previous frame l_lowt-1, and the next frame l_lowt+1, where the previous and next frames l_lowt-1 are supplementary frames. There are 2 motion estimation models: one for the previous frame (ME_1) and one for the next frame (ME_2). The low-light video processing device inputs the target video frame l_lowt and the previous frame l_lowt-1 into the previous frame motion estimation model ME_1 to obtain the motion vector corresponding to the previous frame l_lowt-1. The low-light video processing device inputs the target video frame l_lowt and the next frame l_lowt+1 into the next frame motion estimation model ME_2 to obtain the motion vector corresponding to the next frame l_lowt+1.

[0046] In this 2n motion estimation model, each of the n preceding frame motion estimation models corresponding to the n preceding frames in the frame image set of the target video frame is trained using the video frame in a low-light video segment and the m-th preceding frame of that video frame, according to the m-th second loss function, where m is a positive integer and n ≥ m ≥ 1. The input parameters of the m-th second loss function include the video frame, the m-th preceding frame of that video frame, and the third motion vector corresponding to the m-th preceding frame. Similarly, each of the n following frame motion estimation models corresponding to the n following frames in the frame image set is trained using the video frame and the i-th following frame of that video frame, according to the i-th third loss function, where i is a positive integer and n ≥ i ≥ 1. The input parameters of the i-th third loss function include the video frame, the i-th following frame of that video frame, and the fourth motion vector corresponding to the i-th following frame. For example, the training process for each of the 2n motion estimation models can be essentially the same. This embodiment uses the training process of the m-th previous frame image motion estimation model among n previous frame image motion estimation models corresponding to n previous frame images in a frame image set as an example. The training process includes: inputting a video frame from a low-light video and the m-th previous frame image of that video frame into an initial convolutional neural network model to obtain the third motion vector corresponding to the m-th previous frame image. Then, inputting the video frame, the m-th previous frame image, and the third motion vector into the m-th second loss function to obtain the loss result. If the loss result is not within the function's convergence range, adjusting the network parameters of the initial convolutional neural network model to obtain an adjusted convolutional neural network model. This input operation is repeated until the loss result of the second loss function reaches the function's convergence range, at which point the obtained convolutional neural network model is determined as the m-th previous frame image motion estimation model.

[0047] In one embodiment, taking n=1 as an example, the two motion estimation models include a third previous frame image motion estimation model corresponding to the previous frame image in the frame image set, and a fourth subsequent frame image motion estimation model corresponding to the subsequent frame image. The third previous frame image motion estimation model is obtained by training a video frame in a low-light video segment and a previous frame image of that video frame using a second loss function. This second loss function is Loss... ME_1 satisfy:

[0048] Loss ME_1 =MSE(I_low t W b (I_low t-1 ,f t-1→t ));

[0049] l_low t Indicates a video frame. l_low t-1This represents the frame preceding the current video frame. t represents the time of the current video frame. t-1 represents the time of the frame preceding the current video frame. f t-1→t This represents the motion vector corresponding to the previous frame image determined by the motion estimation model. MSE represents the mean squared error. W b (I_low t-1 ,f t-1→t ) represents the previous frame image determined after motion compensation (e.g., image affine transformation) of the previous frame image based on the motion vectors corresponding to the previous frame image determined by the motion estimation model.

[0050] The motion estimation model for the fourth subsequent frame is trained on the video frame and the next frame using the third loss function. ME_2 satisfy:

[0051] Loss ME_2 =MSE(I_low t W b (I_low t+1 ,f t+1→t ))

[0052] l_low t Indicates a video frame. l_low t+1 This represents the image of the frame following the first frame of the video frame. t+1 represents the time of the frame following the first frame of the video frame. t+1→t This represents the motion vector corresponding to the previous frame image determined by the motion estimation model. MSE represents the mean squared error. W b (I_low t+1 ,f t+1→t ) represents the motion vector corresponding to the subsequent frame image determined based on the motion estimation model, and the subsequent frame image is determined after motion compensation (e.g., image affine transformation) is applied to the subsequent frame image.

[0053] In this way, since the motion estimation models that determine the motion vectors corresponding to each supplementary frame are trained separately, the accuracy of the motion vectors corresponding to each supplementary frame determined based on different motion estimation models is relatively high.

[0054] In the third implementation, the low-light video processing device inputs the target video frame and each preceding frame image from the set of frame images of the target video frame into a preceding frame image motion estimation model to obtain the motion vector corresponding to each preceding frame image. Furthermore, it inputs the target video frame and each subsequent frame image from the set of frame images of the target video frame into a subsequent frame image motion estimation model to obtain the motion vector corresponding to each subsequent frame image.

[0055] In this embodiment, the motion estimation model for the preceding frame image is referred to as the fifth motion estimation model, and the motion estimation model for the following frame image is referred to as the sixth motion estimation model. The fifth motion estimation model is different from the sixth motion estimation model. The fifth motion estimation model can be understood as the motion estimation model for all preceding frames in the corresponding frame image set, and the sixth motion estimation model can be understood as the motion estimation model for all following frames in the corresponding frame image set. The training process for determining the fifth and sixth motion estimation models can refer to the training process of the motion estimation model in the first implementation method described above, or refer to the training process of the motion estimation model in the second implementation method described above; this embodiment will not elaborate on this further.

[0056] The method used in this embodiment to determine the motion vector corresponding to each supplementary frame image in the frame image set of the target video frame should include, but is not limited to, the above three implementation methods. Other methods can also be used, and this embodiment does not limit them.

[0057] Step S220: Based on the motion vector corresponding to each supplementary frame image, perform motion compensation on each supplementary frame image to determine the motion-compensated supplementary frame image.

[0058] A low-light video processing device moves each pixel block (or pixel) in each supplementary frame image based on the motion vector corresponding to each supplementary frame image. In one embodiment, motion compensation (MC) can be achieved through image warping. The low-light video processing device can perform image warping on each supplementary frame image based on the motion vector corresponding to each supplementary frame image in the frame image set of the target video frame to determine the motion-compensated frame image corresponding to each supplementary frame image. For example, the low-light video processing device can move each pixel in the supplementary frame image based on the motion vector corresponding to the supplementary frame image, so that the position of each pixel in the moved supplementary frame image is the original position after being moved according to the motion vector, and the moved supplementary frame image is the motion-compensated supplementary frame image. For example, the low-light video processing device adds the current position of each pixel in the supplementary frame image to the motion vector corresponding to the supplementary frame image to determine the target position of each pixel, and moves each pixel in the supplementary frame image from the current position to the target position.

[0059] Step S230: Concatenate the target video frame and each motion-compensated supplementary frame image by channel to obtain a set of frame images of the target video frame after image alignment.

[0060] In one embodiment, the low-light video processing device concatenates the target video frame and motion-compensated supplementary frame images from its frame image set according to the red (R) channel, green (G) channel, and blue (B) channel to obtain an image-aligned frame image set of the target video frame. This frame image set is then used as input to the subsequent low-light enhancement model. By concatenating the target video frame and each motion-compensated supplementary frame image according to the R, G, and B channels, compared to output methods such as concatenating by image rows and columns, the original feature information of the frame images can be preserved to the greatest extent, without destroying the feature information of the frame images themselves. This results in richer image information in the frame image set subsequently input to the low-light enhancement model, which is beneficial to improving the enhancement effect of the frame images.

[0061] In one embodiment, the low-light enhancement model can be used to obtain the enhanced frame image corresponding to the target video frame based on a set of frame images aligned with the input image. For example, the low-light enhancement model can employ the principles of the Retinex theory. For instance, the low-light enhancement model can be a KinD network model. In one embodiment, the low-light enhancement model can be trained based on a convolutional neural network, and the light enhancement module can be a convolutional neural network model. Figure 4 As shown, this low-light enhancement model (also known as the RetinexNet theoretical network) includes: a decomposition sub-model (DecomNet), a reflection map enhancement sub-model (RestorationNet), and an illumination map enhancement sub-model (AdjustNet). Part of the output data from the decomposition sub-model serves as the input data for the reflection map enhancement sub-model, and the other part of the output data serves as the input data for the illumination map enhancement sub-model. Specifically, the decomposition sub-model decomposes the set of target video frames aligned with the input image into a reflection component map R and an illumination component map L, based on the Retinex principle. The reflection map enhancement sub-model performs denoising, detail enhancement, and brightness boosting on the reflection component map decomposed by the decomposition sub-model to obtain the enhanced reflection component map. The illumination map enhancement sub-model performs brightness boosting on the illumination component map decomposed by the decomposition sub-model to obtain the enhanced illumination component map.

[0062] Please refer to Figure 5 This document illustrates a flowchart of a method for obtaining an enhanced frame image based on a low-light enhancement model, as provided in this embodiment. Figure 5 As shown, the low-light video processing device aligns all frame images in the set of all video frames and then inputs them into a pre-trained low-light enhancement model to obtain the enhanced frame images. The process may include:

[0063] Step S510: Adjust the original resolution of each frame image in the target frame image set of the target video frame after image alignment to a resolution threshold, to obtain the target frame image set with adjusted resolution. This resolution threshold is less than the original resolution.

[0064] Each frame in the target frame image set, after image alignment of all frame images, is the target frame image and supplementary frame images (previous and subsequent frame images) included in the image-aligned target frame image set. In one embodiment, the low-light video processing device performs a size resizing process (resize) on each frame image in the image-aligned target frame image set to obtain a size-resized set of each frame image. This size-resized set of each frame image is the target frame image set with adjusted resolution. The resolution of each size-resized frame image is a resolution threshold, which ranges from 256×256 to 512×512. For example, the resolution threshold is 256×256 or 512×512. For example, the size resizing process can employ a nearest-neighbor interpolation algorithm.

[0065] By reducing the resolution of each frame in the target frame image set after image alignment, the low-light enhancement model can be trained using low-resolution frame images, which improves the training speed of the low-light enhancement model and reduces the number of network parameters.

[0066] Step S520: Input the target frame image set after resolution adjustment into the decomposition sub-model of the low-light enhancement model to obtain the reflection component map and illumination component map corresponding to the target frame image set.

[0067] In one embodiment, the decomposition sub-model of the low-light enhancement model is obtained by training low-light video frames and reference illumination (ground truth) video frames from a low-light video segment using a fourth loss function. The input parameters of the fourth loss function include: the low-light video frame, the reference illumination video frame, the reflectance component map corresponding to the low-light video frame, the illumination component map corresponding to the low-light video frame, the reflectance component map corresponding to the reference illumination video frame, and the illumination component map corresponding to the reference illumination video frame. The reference illumination video frame is a normal illumination video frame, and the illuminance of this video frame is an illumination threshold, which can be 6 illuminance.

[0068] For example, the decomposition sub-model is a convolutional neural network (CNN) model. The training process of this decomposition sub-model may include: inputting low-light video frames and reference-light video frames into an initial CNN model to obtain the reflectance and illumination component maps corresponding to the low-light video frames, and the corresponding reflectance and illumination component maps corresponding to the reference-light video frames. Then, inputting the low-light video frames, reference-light video frames, and their corresponding reflectance and illumination component maps into a fourth loss function to obtain the loss result. If the loss result is not within the function's convergence range, adjusting the network parameters of the initial CNN model to obtain an adjusted CNN model. Repeating the input operation until the loss result of the fourth loss function reaches the function's convergence range, thus confirming the obtained CNN model as the decomposition sub-model.

[0069] In one embodiment, the fourth loss function loss decom satisfy:

[0070]

[0071]

[0072]

[0073]

[0074]

[0075] I low This represents the low-light video frame input to the decomposition submodel. GT This represents the reference illumination video frame input to the decomposed submodel. R low This represents the reflectance component map corresponding to a low-light video frame. L low This represents the illumination component map corresponding to low-light video frames. R GT This represents the reflectance component map corresponding to the reference illumination video frame. GT This represents the illumination component map corresponding to the reference illumination video frame. |·| represents the absolute value. ο represents the dot product operation. ▽ represents the sum of the horizontal and vertical gradients. ||·||1 represents the 1-norm. ||·||2 represents the 2-norm. max represents finding the maximum value. exp represents the exponential operation.

[0076] In this embodiment, as shown Figure 6As shown, the process of obtaining the luminance component map in the decomposition sub-model may include: sequentially performing four convolutional processes (including convolution process 1, convolution process 2, convolution process 3, and convolution process 4) on the target frame image set 610 obtained in step S510 with an adjusted resolution, with a stride parameter of 2. These four convolutional processes are conv3In8Out, conv 8In16Out, conv 16In32Out, and conv32In64Out. Here, conv 3In8Out indicates that the number of input channels for the convolutional process is 3 and the number of output channels is 8; conv 8In16Out indicates that the number of input channels for the convolutional process is 8 and the number of output channels is 16; conv 16In32Out indicates that the number of input channels for the convolutional process is 16 and the number of output channels is 32; and conv32In64Out indicates that the number of input channels for the convolutional process is 32 and the number of output channels is 64. The data from the four convolutional processes are then subjected to global feature extraction and local feature extraction.

[0077] The overall feature extraction process for this sample includes performing 11 convolutions on the data from the 4-times convolutional processing, with each convolution being conv 64In64Out. conv 64In64Out indicates that the number of input channels and output channels for the convolutional processing are both 64. The data after the 11 convolutions is then resized. This first resize operation, resize[1,1024], stretches and flattens the data after the 11 convolutions into a one-dimensional array. The data after the first resize operation is then subjected to three full convolutions (including full convolution 1, full convolution 2, and full convolution 3), which are FC 1024In256Out, FC 256In128Out, and FC 128In64Out. Wherein, FC1024In256Out represents a fully convolutional processing with 1024 input channels and 256 output channels; FC 256In128Out represents a fully convolutional processing with 256 input channels and 128 output channels; and FC 128In64Out represents a fully convolutional processing with 128 input channels and 64 output channels. The data after three fully convolutional processing undergoes a second resize operation. This second resize operation, resize[1,1,1,64], transforms the data after three fully convolutional processing to [1,1,1,64]. The sample local feature extraction process involves performing six convolutional operations on the data after four convolutional operations, each convolution being conv 64In64Out. conv 64In64Out represents a convolutional processing with 64 input channels and 64 output channels. Next, the data after local feature extraction and the data after overall feature extraction are summed at corresponding positions (also known as pixel summation). The sum of corresponding positions, Add[1,16,16,64], indicates that the size after pixel summation becomes [1,16,16,64]. The data after pixel summation is then convolved, conv 64In256Out. The convolved data is then reshaped. This reshape is [1,256,256,1]. The reshaped data is then resized a third time. This resize is [1,256,256,1], ensuring the output size is [1,256,256,1]. This data after third-time reshaping represents the luminance component map 620 of the target video frame.

[0078] Step S530: Adjust the resolution of the reflection component map to the original resolution to obtain a reflection component map with the original resolution. Adjust the resolution of the illumination component map to the original resolution to obtain an illumination component map with the original resolution.

[0079] The low-light video processing device obtains the original resolution of the reflection component map obtained from the decomposed sub-model of the low-light enhancement model. This reflection component map is then resized to obtain a resized reflection component map, the resolution of which is the original resolution. Similarly, an illumination component map with the original resolution is obtained. For example, this resizing process can employ the nearest neighbor interpolation algorithm.

[0080] Step S540: Input the reflection component map with the original resolution into the reflection map enhancement sub-model to obtain the enhanced reflection component map.

[0081] In one embodiment, the reflectance enhancement sub-model of the low-light enhancement model is obtained by training a specific reflectance component map corresponding to a low-light video frame using a fifth loss function. The input parameters of the fifth loss function include: the specific reflectance component map corresponding to a reference light-light video frame and the enhanced reflectance component map obtained by inputting the specific reflectance component map corresponding to the low-light video frame into the reflectance enhancement sub-model. Specifically, the specific reflectance component map corresponding to the low-light video frame is the reflectance component map obtained by inputting the low-light video frame into the trained decomposition sub-model, and the specific reflectance component map corresponding to the reference light-light video frame is the reflectance component map obtained by inputting the reference light-light video frame into the trained decomposition sub-model.

[0082] For example, the reflection map enhancement submodule is a convolutional neural network model. The training process of this reflection map enhancement submodel may include: inputting a specific reflection component map corresponding to a low-light video frame into an initial convolutional neural network model to obtain an enhanced reflection component map; inputting this enhanced reflection component map and a specific reflection component map corresponding to a reference light-light video frame into a fifth loss function to obtain a loss result; if the loss result is not within the function's convergence range, adjusting the network parameters of the initial convolutional neural network model to obtain an adjusted convolutional neural network model; repeating the input operation until the loss result of the fifth loss function reaches the function's convergence range, thus confirming the obtained convolutional neural network model as the reflection map enhancement submodel.

[0083] In one embodiment, the fifth loss function loss Re satisfy:

[0084]

[0085] R GT This represents the reflectance component map corresponding to the reference illumination video frame. R Re Represents the enhanced reflectance map components. SSIM indicates the calculation of structural similarity loss;

[0086] Step S550: Input the illumination component map with the original resolution into the illumination map enhancement sub-model to obtain the enhanced illumination component map.

[0087] In one embodiment, the illumination map enhancement sub-model of the low-light enhancement model is obtained by training a specific illumination component map corresponding to a low-light video frame using a sixth loss function. The input parameters of the sixth loss function include: a specific illumination component map corresponding to a reference illumination video frame, a specific illumination component map corresponding to a low-light video frame, and an enhanced illumination component map obtained by inputting the specific illumination component map corresponding to the low-light video frame into the illumination map enhancement sub-model. Specifically, the specific illumination component map corresponding to the low-light video frame is the illumination component map obtained after inputting the low-light video frame into the trained decomposition sub-model, and the specific illumination component map corresponding to the reference illumination video frame is the illumination component map obtained after inputting the reference illumination video frame into the trained decomposition sub-model.

[0088] For example, the illumination map enhancement submodule is a convolutional neural network model. The training process of this illumination map enhancement submodel may include: inputting the specific illumination component map corresponding to the low-light video frame into the initial convolutional neural network model to obtain the enhanced illumination component map; inputting the enhanced illumination component map, the specific illumination component map corresponding to the reference illumination video frame, and the specific illumination component map corresponding to the low-light video frame into the sixth loss function to obtain the loss result; if the loss result is not within the function's convergence range, adjusting the network parameters of the initial convolutional neural network model to obtain the adjusted convolutional neural network model; repeating the input operation until the loss result of the sixth loss function reaches the function's convergence range, thus confirming the obtained convolutional neural network model as the illumination map enhancement submodel.

[0089] In one embodiment, the sixth loss function is... ad satisfy:

[0090]

[0091] L ad This represents the enhanced illumination component diagram. L low This represents the illumination component map corresponding to low-light video frames. L GT This represents the illumination component map corresponding to the reference illumination video frame.

[0092] It should be noted that in this embodiment, the reflection map enhancement sub-model and the illumination map enhancement sub-model can be trained simultaneously, which reduces the time consumed in training the model.

[0093] Step S560: Multiply the enhanced reflection component map and the enhanced illumination component map to obtain the enhanced frame image corresponding to the target video frame.

[0094] In one embodiment, the low-light video processing device performs a dot product between the enhanced reflection component map and the enhanced illumination component map to determine the enhanced frame image corresponding to the target video frame.

[0095] It should be noted that the low-light video processing device may or may not perform step S510. When the video image processing does not perform step S510, step S530 is also unnecessary. In this case, step S520 can be replaced by aligning the target frame image set of the target video frame in all video frames and inputting it into the decomposition sub-model to obtain the reflection component map and illumination component map corresponding to the target frame image set. Step S540 can be replaced by inputting the reflection component map into the reflection map enhancement sub-model to obtain the enhanced reflection component map. Step S550 can be replaced by inputting the illumination component map into the illumination map enhancement sub-model to obtain the enhanced illumination component map.

[0096] This embodiment fully considers the high correlation and similarity between consecutive video frames in low-light videos. By using a set of image-aligned frames, including the preceding and following frames of the highly correlated current video frame, as input to the low-light enhancement model, this low-light enhancement model, compared to related technologies that use a single frame as input, obtains richer video frame information. This is because the high correlation and similarity between the multiple input frames ensures the accuracy of the acquired video frame information, thus improving the enhancement effect of the low-light enhancement model on video frames.

[0097] For example, such as Figure 7 As shown, this embodiment uses n=1, that is, the set of frame images of the target video frame including the target video frame 710l_lowt, a previous frame image 720l_lowt-1, and a subsequent frame image 730l_lowt+1, and takes the determination of the motion vector corresponding to each supplementary frame image in the set of frame images in the second implementation method described above as an example to illustrate step S120 of this embodiment. The number of motion estimation models is 2, namely, a different motion estimation model ME_1 for the previous frame image and a motion estimation model ME_2 for the subsequent frame image.

[0098] The low-light video processing device inputs the target video frame 710 and the previous frame image 720 into the previous frame image motion estimation model to obtain the motion vector MV1 corresponding to the previous frame image. The low-light video processing device then inputs the target video frame 710 and the subsequent frame image 730 into the subsequent frame image motion estimation model to obtain the motion vector MV2 corresponding to the subsequent frame image. Based on the motion vector MV1, an affine transformation is performed on the previous frame image 720 to determine the motion-compensated previous frame image 740l_lowt'. Based on motion vector MV2, an affine transformation is performed on the subsequent frame image 730 to determine the motion-compensated subsequent frame image 750l_lowt. The low-light video processing device concatenates the motion-compensated preceding frame image 740, the target video frame 710, and the motion-compensated subsequent frame image 750 according to the red, green, and blue channels to obtain a set of frame images 760I_low for the image-aligned target video frame. The set of frame images 760 is then input into the low-light enhancement model to obtain the enhanced frame image 770I_ENt corresponding to the target video frame.

[0099] Step S130: Generate an enhanced video based on the enhanced frame images.

[0100] The low-light video processing device generates an enhanced video based on the enhanced frame image corresponding to each video frame in all video frames, according to the playback sequence of each video frame.

[0101] In summary, the low-light video processing method provided in this embodiment obtains a set of frame images for the current video frame by acquiring the same number of preceding and following frame images for the current video frame, and then iterates through the low-light video to obtain a set of frame images for all video frames. After aligning the set of frame images for all video frames, the method inputs it into a pre-trained low-light enhancement model to obtain enhanced frame images. Based on the enhanced frame images, an enhanced video is generated, achieving low-light video image quality enhancement and improving the video's brightness, color, contrast, and detail. Furthermore, by using a set of aligned frame images (including multiple frames) as input to the low-light enhancement model, this model, compared to low-light enhancement models using single-frame images as input in related technologies, improves the richness of the acquired video frame information while ensuring accuracy due to the high correlation and similarity of the input multiple frame images, thus enhancing the low-light enhancement effect on video frames.

[0102] Please refer to Figure 8This diagram illustrates the structure of a low-light video processing device provided in this embodiment. The functions of each module in this low-light video processing device can be referenced to the steps and implementation methods of the method in this embodiment. The low-light video processing device includes:

[0103] Model 810 is set to acquire the same number of previous and next frame images as the current video frame of the low-light video to obtain the frame image set of the current video frame; and, the low-light video is traversed to obtain the frame image set of all video frames.

[0104] The model processing module 820 is configured to perform image alignment on all frames in the set of frame images of all video frames, and then input them into a pre-trained low-light enhancement model to obtain enhanced frame images.

[0105] The video generation module 830 is configured to generate an enhanced video based on the enhanced frame image.

[0106] In one embodiment, the model processing module 820 is further configured to: use the target video frame in all video frames as a reference frame image to perform motion estimation on each supplementary frame image in the frame image set of the target video frame; sequentially determine the motion vector corresponding to each supplementary frame image in the frame image set of the target video frame, where the supplementary frame image is the previous frame image and the next frame image in the frame image set, and the target video frame is each video frame in all video frames; based on the motion vector corresponding to each supplementary frame image, perform motion compensation on each supplementary frame image to determine the motion-compensated supplementary frame image; and concatenate the target video frame and the motion-compensated supplementary frame images by channel to obtain the frame image set of the target video frame after image alignment.

[0107] In one embodiment, the model processing module 820 is further configured to: for each supplementary frame image in the frame image set of the target video frame, input the target video frame and the supplementary frame image into a pre-trained motion estimation model to obtain the motion vector corresponding to each supplementary frame image.

[0108] In one embodiment, the model processing module 820 is further configured to: input the target video frame and the previous frame image into a pre-trained previous frame image motion estimation model to obtain the motion vector corresponding to the previous frame image; and input the target video frame and the next frame image into a pre-trained next frame image motion estimation model to obtain the motion vector corresponding to the next frame image.

[0109] In one embodiment, the model processing module 820 is further configured to: perform image affine transformation on each supplementary frame image based on the motion vector corresponding to each supplementary frame image, and determine the motion-compensated supplementary frame image.

[0110] In one embodiment, the low-light enhancement model includes a decomposition sub-model, a reflectance enhancement sub-model, and an illumination enhancement sub-model. The model processing module 820 is further configured to: align all frame images in the target frame image set of the target video frame from all video frames, and input them into the decomposition sub-model to obtain the reflectance component map and illumination component map corresponding to the target frame image set, where the target video frame is each video frame in the entire video frame set. The reflectance component map is input into the reflectance enhancement sub-model to obtain the enhanced reflectance component map. The illumination component map is input into the illumination enhancement sub-model to obtain the enhanced illumination component map. The enhanced reflectance component map and the enhanced illumination component map are multiplied together to obtain the enhanced frame image corresponding to the target video frame.

[0111] In one embodiment, the model processing module 820 is further configured to adjust the original resolution of each frame image after image alignment in the target frame image set to a resolution threshold, resulting in a target frame image set with adjusted resolution, where the resolution threshold is less than the original resolution. The target frame image set with adjusted resolution is input to the decomposition sub-model to obtain the reflection component map and illumination component map corresponding to the target frame image set. The resolution of the reflection component map is adjusted to the original resolution, resulting in a reflection component map with the original resolution. The resolution of the illumination component map is adjusted to the original resolution, resulting in an illumination component map with the original resolution. The reflection component map with the original resolution is input to the reflection map enhancement sub-model to obtain an enhanced reflection component map. The illumination component map with the original resolution is input to the illumination map enhancement sub-model to obtain an enhanced illumination component map.

[0112] In one embodiment, the acquisition module 810 is further configured to acquire a frame before and a frame after the current video frame of the low-light video.

[0113] In one embodiment, the acquisition module 810 is further configured to: if the number of the preceding frame image and the following frame image of the current video frame are different, then copy the current video frame as the preceding frame image or the following frame image until the number is the same.

[0114] In summary, the low-light video processing device provided in this embodiment acquires the same number of preceding and following frame images of the current video frame of a low-light video through an acquisition module, obtaining a set of frame images for the current video frame. It then iterates through the low-light video to obtain a set of frame images for all video frames. The model processing module aligns these frame image sets and inputs them into a pre-trained low-light enhancement model to obtain enhanced frame images. The video generation module generates an enhanced video based on these enhanced frame images, achieving low-light video image quality enhancement and improving brightness, color, contrast, and detail. Furthermore, by using an image-aligned set of multiple frames as input to the low-light enhancement model, compared to low-light enhancement models using single-frame images as input in related technologies, this model improves the richness of the acquired video frame information while ensuring accuracy due to the high correlation and similarity of the input multiple frames, thus enhancing the video frame enhancement effect.

[0115] Please refer to Figure 9 This diagram illustrates the structure of a low-light video processing device provided in this embodiment. Figure 9 As shown, the low-light video processing device includes a memory 910, a processor 920, a program stored in the memory 910 and executable on the processor 920, and a data bus 930 for establishing communication between the processor 920 and the memory 910. When the program is executed by the processor 920, it performs the following functions: Figure 1 The steps shown are as follows:

[0116] Step S110: Obtain the same number of previous and next frame images of the current video frame of the low-light video to obtain the frame image set of the current video frame; and traverse the low-light video to obtain the frame image set of all video frames.

[0117] Step S120: Align all frame images in the set of all video frames and input them into the pre-trained low-light enhancement model to obtain the enhanced frame images.

[0118] Step S130: Generate an enhanced video based on the enhanced frame images.

[0119] Or, to achieve the following, such as Figure 2 The steps shown are as follows:

[0120] Step S210: Using the target video frame as a reference frame image in all video frames, perform motion estimation on each supplementary frame image in the frame image set of the target video frame, and sequentially determine the motion vector corresponding to each supplementary frame image in the frame image set of the target video frame. The supplementary frame image consists of the preceding frame image and the following frame image in the frame image set of the target video frame.

[0121] Step S220: Based on the motion vector corresponding to each supplementary frame image, perform motion compensation on each supplementary frame image to determine the motion-compensated supplementary frame image.

[0122] Step S230: Concatenate the target video frame and each motion-compensated supplementary frame image by channel to obtain a set of frame images of the target video frame after image alignment.

[0123] Or, to achieve the following, such as Figure 5 The steps shown are as follows:

[0124] Step S510: Adjust the original resolution of each frame image in the target frame image set of the target video frame after image alignment to a resolution threshold, to obtain the target frame image set with adjusted resolution. This resolution threshold is less than the original resolution.

[0125] Step S520: Input the target frame image set after resolution adjustment into the decomposition sub-model of the low-light enhancement model to obtain the reflection component map and illumination component map corresponding to the target frame image set.

[0126] Step S530: Adjust the resolution of the reflection component map to the original resolution to obtain a reflection component map with the original resolution. Adjust the resolution of the illumination component map to the original resolution to obtain an illumination component map with the original resolution.

[0127] Step S540: Input the reflection component map with the original resolution into the reflection map enhancement sub-model to obtain the enhanced reflection component map.

[0128] Step S550: Input the illumination component map with the original resolution into the illumination map enhancement sub-model to obtain the enhanced illumination component map.

[0129] Step S560: Multiply the enhanced reflection component map and the enhanced illumination component map to obtain the enhanced frame image corresponding to the target video frame.

[0130] In summary, the low-light video processing device provided in this embodiment obtains a set of frame images for the current video frame by acquiring the same number of preceding and following frame images as the current video frame of the low-light video, and then traverses the low-light video to obtain a set of frame images for all video frames of the low-light video. After image alignment of the set of frame images for all video frames, it is input into a pre-trained low-light enhancement model to obtain enhanced frame images. Based on the enhanced frame images, an enhanced video is generated, achieving low-light video image quality enhancement and improving the brightness, color, contrast, and details of the video. Furthermore, by using a set of frame images including multiple frames after image alignment as input to the low-light enhancement model, this low-light enhancement model, compared to low-light enhancement models that use a single frame image as input in related technologies, improves the richness of the acquired video frame information while ensuring the accuracy of the acquired video frame information due to the high correlation and similarity of the input multiple frame images, thereby improving the enhancement effect of the low-light enhancement model on video frames.

[0131] This application also provides a storage medium storing one or more programs, which can be executed by one or more processors to achieve the following: Figure 1 The steps shown are as follows:

[0132] Step S110: Obtain the same number of previous and next frame images of the current video frame of the low-light video to obtain the frame image set of the current video frame; and traverse the low-light video to obtain the frame image set of all video frames.

[0133] Step S120: Align all frame images in the set of all video frames and input them into the pre-trained low-light enhancement model to obtain the enhanced frame images.

[0134] Step S130: Generate an enhanced video based on the enhanced frame images.

[0135] Or, to achieve the following, such as Figure 2 The steps shown are as follows:

[0136] Step S210: Using the target video frame as a reference frame image in all video frames, perform motion estimation on each supplementary frame image in the frame image set of the target video frame, and sequentially determine the motion vector corresponding to each supplementary frame image in the frame image set of the target video frame. The supplementary frame image consists of the preceding frame image and the following frame image in the frame image set of the target video frame.

[0137] Step S220: Based on the motion vector corresponding to each supplementary frame image, perform motion compensation on each supplementary frame image to determine the motion-compensated supplementary frame image.

[0138] Step S230: Concatenate the target video frame and each motion-compensated supplementary frame image by channel to obtain a set of frame images of the target video frame after image alignment.

[0139] Or, to achieve the following, such as Figure 5 The steps shown are as follows:

[0140] Step S510: Adjust the original resolution of each frame image in the target frame image set of the target video frame after image alignment to a resolution threshold, to obtain the target frame image set with adjusted resolution. This resolution threshold is less than the original resolution.

[0141] Step S520: Input the target frame image set after resolution adjustment into the decomposition sub-model of the low-light enhancement model to obtain the reflection component map and illumination component map corresponding to the target frame image set.

[0142] Step S530: Adjust the resolution of the reflection component map to the original resolution to obtain a reflection component map with the original resolution. Adjust the resolution of the illumination component map to the original resolution to obtain an illumination component map with the original resolution.

[0143] Step S540: Input the reflection component map with the original resolution into the reflection map enhancement sub-model to obtain the enhanced reflection component map.

[0144] Step S550: Input the illumination component map with the original resolution into the illumination map enhancement sub-model to obtain the enhanced illumination component map.

[0145] Step S560: Multiply the enhanced reflection component map and the enhanced illumination component map to obtain the enhanced frame image corresponding to the target video frame.

[0146] The low-light video processing method, device, and storage medium provided in this application embodiment acquire the same number of preceding and following frame images of the current video frame of a low-light video through an acquisition module, obtaining a set of frame images for the current video frame. This is then iterated through the low-light video to obtain a set of frame images for all video frames. The model processing module aligns the set of frame images for all video frames and inputs it into a pre-trained low-light enhancement model to obtain enhanced frame images. The video generation module generates an enhanced video based on the enhanced frame images, achieving low-light video image quality enhancement and improving video brightness, color, contrast, and detail. Furthermore, by using an image-aligned set of multiple frames as input to the low-light enhancement model, compared to low-light enhancement models using single-frame images as input in related technologies, this model improves the richness of the acquired video frame information while ensuring accuracy due to the high correlation and similarity of the input multiple frames, thereby enhancing the video frame enhancement effect of the low-light enhancement model.

[0147] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0148] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0149] The preferred embodiments of this application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of this application shall be within the scope of the claims.

Claims

1. A low-light video processing method, characterized in that, The method includes: Obtain the same number of preceding and following frame images of the current video frame of the low-light video to obtain the frame image set of the current video frame; and traverse the low-light video to obtain the frame image set of all video frames; wherein, the frame image set of any video frame in the all video frames includes that video frame, n preceding frame images of that video frame, and n following frame images of that video frame, where n is a positive integer and n>1; After aligning all frame images in the set of all video frames, the images are input into a pre-trained low-light enhancement model to obtain the enhanced frame images. Based on the enhanced frame images, an enhanced video is generated; The low-light enhancement model includes: a decomposition sub-model, a reflectance enhancement sub-model, and a illumination enhancement sub-model; The process of aligning all frame images in the set of all video frames and then inputting them into a pre-trained low-light enhancement model to obtain enhanced frame images includes: After aligning all frame images in the target frame image set of the target video frame in all video frames, the original resolution of each frame image after image alignment is adjusted to a resolution threshold to obtain the target frame image set after resolution adjustment, wherein the resolution threshold is less than the original resolution; The target frame image set after resolution adjustment is input into the decomposition sub-model to obtain the reflection component map and illumination component map corresponding to the target frame image set, wherein the target video frame is each video frame in all video frames; The resolution of the reflection component map is adjusted to the original resolution to obtain a reflection component map with the original resolution; the resolution of the illumination component map is adjusted to the original resolution to obtain an illumination component map with the original resolution. The reflection component map with the original resolution is input into the reflection map enhancement sub-model to obtain the enhanced reflection component map; The illumination component map with the original resolution is input into the illumination map enhancement sub-model to obtain the enhanced illumination component map. The enhanced reflection component map and the enhanced illumination component map are multiplied together to obtain the enhanced frame image corresponding to the target video frame.

2. The method according to claim 1, characterized in that, The step of aligning all frame images in the set of frame images of all video frames includes: Using the target video frame in all video frames as a reference frame image, motion estimation is performed on each supplementary frame image in the frame image set of the target video frame. The motion vector corresponding to each supplementary frame image in the frame image set of the target video frame is determined sequentially. The supplementary frame image is the previous frame image and the next frame image in the frame image set. The target video frame is each video frame in all video frames. Based on the motion vector corresponding to each supplementary frame image, motion compensation is performed on each supplementary frame image to determine the motion-compensated supplementary frame image. The target video frame and the motion-compensated supplementary frame images are concatenated by channel to obtain a set of frame images of the target video frame after image alignment.

3. The method according to claim 2, characterized in that, The step of using the target video frame as a reference frame image in all video frames to perform motion estimation on each supplementary frame image in the frame image set of the target video frame, and sequentially determining the motion vector corresponding to each supplementary frame image in the frame image set of the target video frame, includes: For each supplementary frame image in the frame image set of the target video frame, the target video frame and the supplementary frame image are input into a pre-trained motion estimation model to obtain the motion vector corresponding to the supplementary frame image.

4. The method according to claim 2, characterized in that, The step of using the target video frame as a reference frame image in all video frames to perform motion estimation on each supplementary frame image in the frame image set of the target video frame, and sequentially determining the motion vector corresponding to each supplementary frame image in the frame image set of the target video frame, includes: The target video frame and the previous frame image are input into a pre-trained motion estimation model for the previous frame image to obtain the motion vector corresponding to the previous frame image. The target video frame and the subsequent frame image are input into a pre-trained motion estimation model for the subsequent frame image to obtain the motion vector corresponding to the subsequent frame image.

5. The method according to claim 2, characterized in that, The step of performing motion compensation on each supplementary frame image based on the motion vector corresponding to each supplementary frame image to determine the motion-compensated supplementary frame image includes: Based on the motion vector corresponding to each supplementary frame image, an image affine transformation is performed on each supplementary frame image to determine the motion-compensated supplementary frame image.

6. The method according to any one of claims 1-5, characterized in that, The process of acquiring the same number of previous and subsequent frame images as the current video frame of the low-light video includes: Get the previous frame and the next frame of the current video frame in a low-light video.

7. The method according to any one of claims 1-5, characterized in that, Before acquiring the same number of previous and subsequent frame images as the current video frame of the low-light video, the method further includes: If the number of preceding and following frames of the current video frame is different, then the current video frame is copied as either the preceding or following frame until the number is the same.

8. A low-light video processing device, characterized in that, The device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the steps of the low-light video processing method as described in any one of claims 1 to 7.

9. A storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the low-light video processing method according to any one of claims 1 to 7.

Images