Low-illumination video frequency division enhancement method based on Laplace decomposition

Through the low-illumination video frequency division enhancement method based on Laplace decomposition, the low-frequency signal and high-frequency details of the image are decoupled, and the low-illumination video is enhanced frame by frame, solving the problems of oversmooth details and color offset in the prior art, and improving the Video details rebuild quality and color accuracy.

CN119006312AActive Publication Date: 2024-11-22JINHUA INSTITUTE OF ZHEJIANG UNIVERSITY

Patent Information

Application Number
CN202411029020.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-22
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

The existing low-illumination video enhancement methods have excessive details and color shifts while improving image visibility, which affects the video image quality and visual experience.

Method used

Using the low-illumination video frequency division enhancement method based on Laplace decomposition, a low-illumination video enhancement model including the image multi-scale decomposition operator, multi-scale enhancement network and image multi-scale reconstruction operator is established. The reconstruction loss function and the time domain consistency loss function are used for training, decoupling the image low-frequency signal and high-frequency details, and enhance low-illumination video frame by frame.

Benefits of technology

It effectively improves the quality of detail reconstruction and color accuracy of low-illumination videos, improves oversmoothing and color shifting, and achieves better visual effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006312A_ABST
    Figure CN119006312A_ABST
Patent Text Reader

Abstract

The invention discloses a low-illumination video frequency division enhancement method based on Laplace decomposition. The method comprises the following steps: establishing a low-illumination video enhancement model comprising an image multi-scale decomposition operator and a reconstruction operator based on a Laplacian pyramid and a multi-scale enhancement network, and constructing an overall loss function of the model, wherein the overall loss function comprises a reconstruction loss function and a time domain consistency loss function; training the model until the convergence training of the overall loss function is completed; and inputting video frames of a to-be-enhanced low-illumination video into the model for processing, and then outputting video frames enhanced frame by frame. According to the method, the problems of low video visibility and low signal-to-noise ratio can be solved, the model and the loss function constructed by the method can decouple low-frequency signals and high-frequency details of the image during frame-by-frame enhancement, the low-illumination image can be recovered and enhanced, and finally the image with the original size is reconstructed. According to the method, the low-illumination video can be enhanced frame by frame, the detail reconstruction quality and the color accuracy of the enhancement of the low-illumination video are effectively improved, and a better visual effect can be obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a video frequency division enhancement method and relates to the field of image processing, in particular to a low-illumination video frequency division enhancement method based on Laplace decomposition. Background Art

[0002] Low-light video enhancement technology is usually used to solve the problems of low video visibility and low signal-to-noise ratio caused by insufficient lighting, uneven lighting, backlight and other environmental conditions. Traditional methods based on histogram equalization and image decomposition models will amplify noise while improving the overall brightness of the image. With the development of deep learning technology, the low-light enhancement method of convolutional neural network can suppress noise while improving image visibility. Since frame-by-frame paired low-light and regular-light videos are difficult to obtain, early low-light video enhancement tasks are often completed through single-image enhancement networks. However, such methods often have flickering in video enhancement tasks, affecting visual perception.

[0003] To solve the above problems, some data sets and data simulation methods for training low-light video enhancement network models have been proposed one after another, such as the first public low-light video enhancement data set "Dark Raw Video Dataset (DRV)", which contains low-light and regular-light video pairs of static scene videos, and the dynamic low-light video enhancement data set "Seeing Dynamic Scenes in the Dark (SDSD)" based on the slide rail system. In addition, some methods for generating simulated dynamic low-light videos from regular video data sets have also been gradually proposed. The emergence of low-light video data sets and simulation methods has prompted researchers to develop network models for low-light video enhancement, such as SMID, SDSDNet, and LVE-S2D models. However, the existing methods have different degrees of detail over-smoothing and color cast, which seriously affects the video image quality and visual experience. Summary of the invention

[0004] In order to solve the problems existing in the background technology, the present invention provides a low-illumination video frequency division enhancement method based on Laplace decomposition.

[0005] The technical solution adopted by the present invention is:

[0006] The low-illumination video frequency division enhancement method based on Laplace decomposition of the present invention comprises:

[0007] S1: Establish a low-light video enhancement model including a Laplacian pyramid-based image multi-scale decomposition operator, a multi-scale enhancement network and a Laplacian pyramid-based image multi-scale reconstruction operator connected in sequence, and construct an overall loss function of the low-light video enhancement model including a reconstruction loss function and a time domain consistency loss function; the ambient light illumination below 0.5lux is regarded as low illumination.

[0008] S2: In different scenarios, several low-illumination RGB videos and their reference RGB videos are collected and divided into video frames. The video frames of each low-illumination RGB video and its reference RGB video are input into the low-illumination video enhancement model for training until the overall loss function converges, thereby obtaining a trained low-illumination video enhancement model.

[0009] S3: Input the video frames of the low-illumination RGB video to be enhanced into the trained low-illumination video enhancement model for processing. After processing, the trained low-illumination video enhancement model outputs the enhanced video frames frame by frame to achieve frequency division enhancement of the low-illumination video.

[0010] The low-light video enhancement model uses an image multi-scale decomposition operator to decompose the image into low-frequency components and multi-scale detail components; the low-frequency components and detail components are input into a multi-scale enhancement network for frame-by-frame enhancement; and the image reconstruction operator is used to reconstruct the high-quality enhancement result of the original size. The method of the present invention uses a Laplacian pyramid to perform multi-scale decomposition of the low-frequency components and detail components of the image, suppresses noise and improves the detail reconstruction effect of the video.

[0011] In the step 1), the image multi-scale decomposition operator based on the Laplacian pyramid includes N layers of Laplacian pyramids, each layer of the Laplacian pyramid includes two Gaussian low-pass filters G, a 2×2 mean downsampling layer D 2×2 and a 2×2 bilinear interpolation upsampling layer U 2×2 , the input of the i-th layer of the Laplacian pyramid passes through its first Gaussian low-pass filter and the 2×2 mean downsampling layer D 2×2 The processed output is taken as the ith intermediate result, and the ith intermediate result is successively upsampled by its own 2×2 bilinear interpolation layer U 2×2The output after processing with the second Gaussian low-pass filter G is used as the i-th decomposition result, the i-th intermediate result and the i+1-th decomposition result are jointly processed by pixel-by-pixel subtraction and output as the i-th detail component, N=1, 2, ..., i, ..., N-1, wherein the input of the first layer of the Laplacian pyramid is the low-illumination video frame of the input low-illumination video enhancement model, the low-illumination video frame and the first decomposition result are jointly processed by pixel-by-pixel subtraction and output as the bottom layer detail component, the i-th intermediate result of the i-th layer of the Laplacian pyramid is directly used as the low-frequency component of the top layer, and finally the obtained low-frequency component of the top layer, N-1 i-th detail components and the bottom layer detail components are jointly used as the multi-scale decomposition result of the low-illumination image and input into the multi-scale enhancement network for multi-scale enhancement processing.

[0012] In the step 1), the multi-scale enhancement network includes N+1 layers of encoding and decoding network layers, each layer of the encoding and decoding network layer includes three encoding network blocks, an intermediate network block and three decoder network blocks connected in sequence, and the input of each layer of the encoding and decoding network layer is sequentially input into the three encoding network blocks, the intermediate network blocks and the three decoder network blocks for processing, the output of the first encoding network block jumps to the third decoder network block and is processed together with the output of the second decoder network block, the output of the second encoding network block jumps to the second decoder network block and is processed together with the output of the first decoder network block, the output of the third encoding network block jumps to the first decoder network block and is processed together with the output of the intermediate network block, and the output of the third decoder network block is used as the output of the multi-scale enhancement network; the low-frequency components of the top layer, the N-1 th i-th detail components and the bottom layer detail components in the multi-scale decomposition result of the low-illumination image are respectively input into their respective encoding and decoding network layers for processing, and the respective multi-scale enhancement results are output after processing. The multi-scale decomposition true value video frame is used to supervise each multi-scale enhancement result, and the supervision signal is a loss function.

[0013] The encoding network block EB includes two 3×3 convolutional layers with residual connections and a 2×2 convolutional downsampling layer. The input of the encoding network block EB is sequentially input into the two 3×3 convolutional layers with residual connections for processing to obtain the convolution result. The convolution result and the input of the encoding network block EB are added element by element and then processed by the 2×2 convolutional downsampling layer and output as the output of the encoding network block EB.

[0014] The intermediate network block MB includes two 3×3 convolutional layers with residual connections, a fully connected layer and two channel attention modules. The input of the intermediate network block MB is sequentially input into the two 3×3 convolutional layers with residual connections for processing. The processed output is then input into the first channel attention module together with the input of the intermediate network block MB for processing. The output of the first channel attention module passes through the fully connected layer and then through the channel weight vector processing, and then is multiplied by the channel dimension with the output of the first channel attention module. The multiplication result is then input into the second channel attention module for processing and the output is output as the output of the intermediate network block MB.

[0015] The decoder network block DB includes four 3×3 convolutional layers with residual connections, a 1×1 convolutional layer, a feature splicing operation Cat and an upsampling operation pixel shuffle. The input of the decoder network block DB is sequentially input into two 3×3 convolutional layers with residual connections, a 1×1 convolutional layer and another two 3×3 convolutional layers with residual connections for processing. The output of the 1×1 convolutional layer and the output of the fourth 3×3 convolutional layer with residual connections are processed together after the feature splicing operation Cat and the upsampling operation pixel shuffle. The final output is used as the output of the decoder network block DB.

[0016] In the step 1), the image multi-scale reconstruction operator based on the Laplacian pyramid includes N layers of Laplacian pyramid reconstruction layers, each of which includes a Gaussian low-pass filter and a 2×2 bilinear interpolation upsampling layer U connected in sequence. 2×2 The output of the i-th detail component after multi-scale enhancement processing and the output of the i+1-th Laplacian pyramid reconstruction layer is processed together using pixel-by-pixel addition as the input of the i-th Laplacian pyramid reconstruction layer, the low-frequency component of the top layer after multi-scale enhancement processing is used as the input of the i-th Laplacian pyramid reconstruction layer, and the output of the bottom layer detail component after multi-scale enhancement processing and the output of the first Laplacian pyramid reconstruction layer is processed together using pixel-by-pixel addition as the output of the image multi-scale reconstruction operator based on the Laplacian pyramid, that is, as the original size enhancement result of the low-illumination video enhancement model.

[0017] In step 1), the loss function L is reconstructed rec The details are as follows:

[0018]

[0019] in, and They represent the i-th multi-scale enhancement result after the multi-scale enhancement network processes the video frame of the low-illumination RGB video and the i-th multi-scale decomposition true value video frame of the reference RGB video; L perrepresents the perceptual loss, λ per Denotes the perceptual loss L per The weight coefficient of .

[0020] The perceptual loss L per The details are as follows:

[0021]

[0022] Among them, F VGG (·) indicates the multi-layer features extracted from the video frame using the pre-trained deep convolutional neural network VGG (Very Deep Convolutional Networks for Large-scale Image Recognition); and They respectively represent the i-th multi-scale enhancement result after the multi-scale enhancement network processes the video frame of the low-illumination RGB video and the i-th multi-scale decomposition true value video frame of the video frame of the reference RGB video; Represents the 1-norm.

[0023] In the step 1), the temporal consistency loss function L tc The details are as follows:

[0024]

[0025] Among them, N p It represents the total number of image blocks of size k×k into which the multi-scale enhancement results of two consecutive video frames are divided; and They represent the pixel mean of the j-th k×k image block respectively; Represents the 1-norm.

[0026] The method of the present invention first starts from the frequency and characteristics of the neural network model, and uses a multi-scale decomposition enhancement method to enhance low-light video frame by frame. By multi-channel enhancement of multi-scale base and detail, the network is promoted to fit high-frequency details and low-frequency content (such as color signals) respectively, which not only improves the over-smoothing phenomenon, but also improves color deviation. Considering the large error of optical flow estimation in low-light images, this method further suppresses flicker through dynamic temporal consistency loss, and realizes explicit temporal consistency constraints without optical flow on dynamic data sets; compared with existing algorithms, this method effectively improves the spatial reconstruction quality and color accuracy of low-light video enhancement, and can achieve better visual effects.

[0027] The beneficial effects of the present invention are:

[0028] The method of the present invention solves the problems of low video visibility and low signal-to-noise ratio caused by environmental factors such as insufficient lighting, uneven lighting, and backlight during video shooting. Taking into account the differences in the fitting speed of neural networks for signals of different frequencies, the method constructs a low-light video enhancement model based on Laplace pyramid decomposition, decouples the low-frequency signals and high-frequency details of the image during frame-by-frame enhancement, constructs a multi-scale enhancement network to restore and enhance low-light images, and finally reconstructs the original size image signal. The network is trained using a reconstruction loss function and a time-domain consistency loss function. Compared with existing methods, the method can enhance low-light videos frame by frame, effectively improves the detail reconstruction quality and color accuracy of low-light video enhancement, and can achieve better visual effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the image multi-scale decomposition operator of the method of the present invention;

[0030] Figure 2 is a schematic diagram of a multi-scale enhancement network of the method of the present invention;

[0031] Figure 3 Schematic diagram of the image multi-scale reconstruction operator of the method of the present invention;

[0032] Figure 4 It is a schematic diagram of the structure of the encoding network block and the decoding network block of the method of the present invention;

[0033] Figure 5 It is a schematic diagram of the intermediate network block structure of the method of the present invention;

[0034] Figure 6 is a display diagram of a low-light video frame input of an embodiment of the present invention, an enhancement result of the method of the present invention, an enhancement amplification result of the method of the present invention, and an enhancement result of the ChebyLighter model, wherein: Figure 6 (a) is a display diagram of a low-light video frame input of scene 1 of an embodiment of the present invention, an enhancement result of the method of the present invention, an enhancement magnification result of the method of the present invention, and an enhancement result using the ChebyLighter model, Figure 6 (b) is a display diagram of the low-light video frame input of scene 2 of an embodiment of the present invention, the enhancement result of the method of the present invention, the enhancement and magnification result of the method of the present invention, and the enhancement result using the ChebyLighter model, Figure 6 (c) is a display diagram of the low-illumination video frame input of scene three of an embodiment of the present invention, the enhancement result of the method of the present invention, the enhancement and magnification result of the method of the present invention, and the enhancement result using the ChebyLighter model. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] The low-light video frequency division enhancement method based on Laplace decomposition of the present invention is specifically as follows:

[0037] S1: Establish a low-light video enhancement model including a Laplacian pyramid-based image multi-scale decomposition operator, a multi-scale enhancement network and a Laplacian pyramid-based image multi-scale reconstruction operator connected in sequence, and construct an overall loss function of the low-light video enhancement model including a reconstruction loss function and a time domain consistency loss function; the ambient light illumination below 0.5lux is regarded as low illumination.

[0038] like Figure 1 As shown, the image multi-scale decomposition operator based on the Laplacian pyramid includes N layers of Laplacian pyramids, each layer of the Laplacian pyramid includes two Gaussian low-pass filters G, a 2×2 mean downsampling layer D 2×2 and a 2×2 bilinear interpolation upsampling layer U 2×2 , the input of the i-th layer of the Laplacian pyramid passes through its first Gaussian low-pass filter and the 2×2 mean downsampling layer D 2×2 The processed output is taken as the i-th intermediate result, and the i-th intermediate result is successively upsampled by its own 2×2 bilinear interpolation layer U 2×2 The output after being processed by the second Gaussian low-pass filter G is used as the i-th decomposition result, the i-th intermediate result and the i+1-th decomposition result are processed by pixel-by-pixel subtraction and output as the i-th detail component, N=1, 2, ..., i, ..., N-1, wherein the input of the first layer of the Laplacian pyramid is the low-light video frame of the input low-light video enhancement model, the low-light video frame and the first decomposition result are processed by pixel-by-pixel subtraction and output as the bottom layer detail component, the i-th intermediate result of the i-th layer of the Laplacian pyramid is directly used as the low-frequency component of the top layer, and finally the obtained low-frequency component of the top layer, N-1 i-th detail components and the bottom layer detail components are used as the multi-scale decomposition result of the low-light image and input into the multi-scale enhancement network for multi-scale enhancement processing. In specific implementation, a two-layer Laplacian pyramid can be used.

[0039] like Figure 2As shown, the multi-scale enhancement network includes N+1 layers of encoding and decoding network layers, each layer of encoding and decoding network layers includes three encoding network blocks, intermediate network blocks and three decoder network blocks connected in sequence, and the input of each layer of encoding and decoding network layers is sequentially input into three encoding network blocks, intermediate network blocks and three decoder network blocks for processing, the output of the first encoding network block jumps to the third decoder network block and is processed together with the output of the second decoder network block, the output of the second encoding network block jumps to the second decoder network block and is processed together with the output of the first decoder network block, the output of the third encoding network block jumps to the first decoder network block and is processed together with the output of the intermediate network block, and the output of the third decoder network block is used as the output of the multi-scale enhancement network; the low-frequency components of the top layer, the N-1 th i-th detail components and the bottom layer detail components in the multi-scale decomposition result of the low-illumination image are respectively input into their respective encoding and decoding network layers for processing, and the respective multi-scale enhancement results are output after processing. The multi-scale decomposition true value video frame is used to supervise each multi-scale enhancement result, and the supervision signal is a loss function. In specific implementation, a three-layer encoding and decoding network layer can be used.

[0040] like Figure 4 As shown in the figure, the encoding network block EB includes two 3×3 convolutional layers with residual connections and a 2×2 convolutional downsampling layer. The input of the encoding network block EB is sequentially input into the two 3×3 convolutional layers with residual connections for processing to obtain the convolution result. The convolution result and the input of the encoding network block EB are added element by element and then processed by the 2×2 convolutional downsampling layer and output as the output of the encoding network block EB.

[0041] like Figure 5 As shown, the intermediate network block MB includes two 3×3 convolutional layers with residual connections, a fully connected layer and two channel attention modules. The input of the intermediate network block MB is sequentially input into the two 3×3 convolutional layers with residual connections for processing. The processed output is then input into the first channel attention module together with the input of the intermediate network block MB for processing. The output of the first channel attention module passes through the fully connected layer and then through the channel weight vector processing, and then is multiplied by the channel dimension with the output of the first channel attention module. The multiplication result is then input into the second channel attention module and the processed output is used as the output of the intermediate network block MB.

[0042] like Figure 4As shown in the figure, the decoder network block DB includes four 3×3 convolutional layers with residual connections, a 1×1 convolutional layer, a feature splicing operation Cat and an upsampling operation pixel shuffle. The input of the decoder network block DB is sequentially input to two 3×3 convolutional layers with residual connections, a 1×1 convolutional layer and another two 3×3 convolutional layers with residual connections for processing. The output of the 1×1 convolutional layer and the output of the fourth 3×3 convolutional layer with residual connections are processed together after the feature splicing operation Cat and the upsampling operation pixel shuffle is performed, and the final output is used as the output of the decoder network block DB.

[0043] like Figure 3 As shown in FIG. 1 , the image multi-scale reconstruction operator based on the Laplacian pyramid includes N layers of Laplacian pyramid reconstruction layers, each of which includes a Gaussian low-pass filter and a 2×2 bilinear interpolation upsampling layer U connected in sequence. 2×2 The output of the i-th detail component after multi-scale enhancement and the output of the i+1-th Laplacian pyramid reconstruction layer is processed by pixel-by-pixel addition as the input of the i-th Laplacian pyramid reconstruction layer, the low-frequency component of the top layer after multi-scale enhancement is used as the input of the i-th Laplacian pyramid reconstruction layer, and the output of the bottom detail component after multi-scale enhancement and the output of the first Laplacian pyramid reconstruction layer is processed by pixel-by-pixel addition as the output of the image multi-scale reconstruction operator based on the Laplacian pyramid, that is, as the original size enhancement result of the low-illumination video enhancement model. In specific implementation, two layers of Laplacian pyramid reconstruction layers can be used.

[0044] Reconstruction loss function L rec The details are as follows:

[0045]

[0046] in, and They represent the i-th multi-scale enhancement result after the multi-scale enhancement network processes the video frame of the low-illumination RGB video and the i-th multi-scale decomposition true value video frame of the reference RGB video; L per represents the perceptual loss, λ per Denotes the perceptual loss L per The weight coefficient of .

[0047] Perceptual loss L per The details are as follows:

[0048]

[0049] Among them, F VGG (·) represents the multi-layer features extracted from the video frame using the pre-trained deep convolutional neural network VGG; and They respectively represent the i-th multi-scale enhancement result after the multi-scale enhancement network processes the video frame of the low-illumination RGB video and the i-th multi-scale decomposition true value video frame of the video frame of the reference RGB video; Represents the 1-norm.

[0050] Temporal consistency loss function L tc The details are as follows:

[0051]

[0052] Where Np represents the total number of image blocks of size k×k into which the multi-scale enhancement results of two consecutive video frames are divided; and They represent the pixel mean of the j-th k×k image block respectively; Represents the 1-norm.

[0053] S2: In different scenarios, several low-illumination RGB videos and their reference RGB videos are collected and divided into video frames. The video frames of each low-illumination RGB video and its reference RGB video are input into the low-illumination video enhancement model for training until the overall loss function converges, thereby obtaining a trained low-illumination video enhancement model.

[0054] S3: Input the video frames of the low-illumination RGB video to be enhanced into the trained low-illumination video enhancement model for processing. After processing, the trained low-illumination video enhancement model outputs the enhanced video frames frame by frame to achieve frequency division enhancement of the low-illumination video.

[0055] The low-light video enhancement model uses an image multi-scale decomposition operator to decompose the image into low-frequency components and multi-scale detail components; the low-frequency components and detail components are input into a multi-scale enhancement network for frame-by-frame enhancement; and the image reconstruction operator is used to reconstruct the high-quality enhancement result of the original size. The method of the present invention uses a Laplacian pyramid to perform multi-scale decomposition of the low-frequency components and detail components of the image, suppresses noise and improves the detail reconstruction effect of the video.

[0056] The present invention constructs a low-illumination video enhancement model based on Laplace pyramid decomposition, decouples image low-frequency signals and high-frequency details during frame-by-frame enhancement, constructs a multi-scale enhancement network to restore and enhance low-illumination images, and finally reconstructs the original size image signal.

[0057] like Figure 6 (a) Figure 6 (b) and Figure 6(c) is the operating result of the method of the present invention and the ChebyLighter model on the DRV dataset in a specific embodiment of the present invention. It can be seen that the method starts from the frequency and characteristics of the neural network model, adopts a multi-scale decomposition enhancement method to enhance the low-light video frame by frame, and promotes the network to fit high-frequency details and low-frequency content (such as color signals) respectively through multi-channel enhancement of multi-scale base and detail, which effectively improves the spatial reconstruction quality and color accuracy of low-light video enhancement and achieves better visual effects.

Claims

1. A low-light video frequency division enhancement method based on Laplace decomposition, It is characterized in that include: S1: Establish a low-light video enhancement model including a Laplacian pyramid-based image multi-scale decomposition operator, a multi-scale enhancement network, and a Laplacian pyramid-based image multi-scale reconstruction operator connected in sequence, and construct an overall loss function of the low-light video enhancement model including a reconstruction loss function and a temporal consistency loss function; S2: In different scenarios, several low-light RGB videos and their reference RGB videos are collected and divided into video frames, and the video frames of each low-light RGB video and its reference RGB video are input into the low-light video enhancement model for training until the overall loss function converges, thereby obtaining a trained low-light video enhancement model; S3: Input the video frames of the low-illumination RGB video to be enhanced into the trained low-illumination video enhancement model for processing. After processing, the trained low-illumination video enhancement model outputs the enhanced video frames frame by frame to achieve frequency division enhancement of the low-illumination video.

2. The low-light video frequency division enhancement method based on Laplace decomposition according to claim 1, Features: In the step 1), the image multi-scale decomposition operator based on the Laplacian pyramid includes N layers of Laplacian pyramids, each layer of the Laplacian pyramid includes two Gaussian low-pass filters G, a 2×2 mean downsampling layer D 2×2 and a 2×2 bilinear interpolation upsampling layer U 2×2 , the input of the i-th layer of the Laplacian pyramid passes through its first Gaussian low-pass filter and the 2×2 mean downsampling layer D 2×2 The processed output is taken as the i-th intermediate result, and the i-th intermediate result is successively upsampled by its own 2×2 bilinear interpolation layer U 2×2 The output after processing with the second Gaussian low-pass filter G is used as the i-th decomposition result, the i-th intermediate result and the i+1-th decomposition result are jointly processed by pixel-by-pixel subtraction and output as the i-th detail component, N=1, 2, ..., i, ..., N-1, wherein the input of the first layer of the Laplacian pyramid is the low-illumination video frame of the input low-illumination video enhancement model, the low-illumination video frame and the first decomposition result are jointly processed by pixel-by-pixel subtraction and output as the bottom layer detail component, the i-th intermediate result of the i-th layer of the Laplacian pyramid is directly used as the low-frequency component of the top layer, and finally the obtained low-frequency component of the top layer, N-1 i-th detail components and the bottom layer detail components are jointly used as the multi-scale decomposition result of the low-illumination image and input into the multi-scale enhancement network for multi-scale enhancement processing.

3. The low-light video frequency division enhancement method based on Laplace decomposition according to claim 2, Features: In the step 1), the multi-scale enhancement network includes N+1 layers of encoding and decoding network layers, each layer of the encoding and decoding network layer includes three encoding network blocks, an intermediate network block and three decoder network blocks connected in sequence, the input of each layer of the encoding and decoding network layer is sequentially input into the three encoding network blocks, the intermediate network block and the three decoder network blocks for processing, the output of the first encoding network block jumps to the third decoder network block and is processed together with the output of the second decoder network block, the output of the second encoding network block jumps to the second decoder network block and is processed together with the output of the first decoder network block, the output of the third encoding network block jumps to the first decoder network block and is processed together with the output of the intermediate network block, and the output of the third decoder network block is used as the output of the multi-scale enhancement network; the low-frequency components of the top layer, the N-1 i-th detail components and the bottom layer detail components in the multi-scale decomposition result of the low-illumination image are respectively input into their respective encoding and decoding network layers for processing, and their respective multi-scale enhancement results are output after processing.

4. The low-light video frequency division enhancement method based on Laplace decomposition according to claim 2, Features: In the step 1), the image multi-scale reconstruction operator based on the Laplacian pyramid includes N layers of Laplacian pyramid reconstruction layers, each of which includes a Gaussian low-pass filter and a 2×2 bilinear interpolation upsampling layer U connected in sequence. 2×2 The output of the i-th detail component after multi-scale enhancement processing and the output of the i+1-th Laplacian pyramid reconstruction layer is processed together using pixel-by-pixel addition as the input of the i-th Laplacian pyramid reconstruction layer, the low-frequency component of the top layer after multi-scale enhancement processing is used as the input of the i-th Laplacian pyramid reconstruction layer, and the output of the bottom layer detail component after multi-scale enhancement processing and the output of the first Laplacian pyramid reconstruction layer is processed together using pixel-by-pixel addition as the output of the image multi-scale reconstruction operator based on the Laplacian pyramid, that is, as the original size enhancement result of the low-illumination video enhancement model.

5. The low-light video frequency division enhancement method based on Laplace decomposition according to claim 3, Features: In step 1), the loss function L is reconstructed rec The details are as follows: in, and They represent the i-th multi-scale enhancement result after the multi-scale enhancement network processes the video frame of the low-illumination RGB video and the i-th multi-scale decomposition true value video frame of the video frame of the reference RGB video respectively; L per represents the perceptual loss, λ per Denotes the perceptual loss L per The weight coefficient of .

6. The low-light video frequency division enhancement method based on Laplace decomposition according to claim 5, Features: The perceptual loss L per The details are as follows: Among them, F VGG (·) represents the multi-layer features extracted from the video frame using the pre-trained deep convolutional neural network VGG; and They respectively represent the i-th multi-scale enhancement result after the multi-scale enhancement network processes the video frame of the low-illumination RGB video and the i-th multi-scale decomposition true value video frame of the video frame of the reference RGB video; Represents the 1-norm.

7. The low-light video frequency division enhancement method based on Laplace decomposition according to claim 1, Features: In the step 1), the temporal consistency loss function L tc The details are as follows: Among them, N p It represents the total number of image blocks of size k×k into which the multi-scale enhancement results of two consecutive video frames are divided; and They represent the pixel mean of the j-th k×k image block respectively; Represents the 1-norm.

Citation Information

Patent Citations

  • Low-illumination image enhancement method based on adaptive frequency decomposition and related equipment

    CN115063318A

  • Low-light image enhancement method and device fusing high and low frequency feature information

    CN116152120A

  • Low-illumination video enhancement method and system

    CN116703793A

  • Target detection method under low illumination based on pyramid enhanced network

    CN117058019A

  • Low-illumination image enhancement method based on deep learning and Laplacian pyramid

    CN117196968A

Cited By

  • Underwater image dynamic enhancement method based on pyramid network and application

    CN119599924A