Video image super-wide dynamic processing method

Abstract

The invention discloses a video image super-wide dynamic processing method, which comprises the following steps of: firstly, constructing a CNN-LSTM (Convolutional Neural Network-Long Short Term Memory) model consisting of a CNN part and an LSTM part, the CNN part being used for extracting spatial features of each frame; inputting the features extracted by the CNN part into the LSTM part to extract dynamic information of the features between frames; the output of the LSTM part is sent to a full connection layer for processing, and then a video image after wide dynamic processing is generated; collecting video image data under different exposure settings and dividing the video image data into a training set and a test set; decomposing image data in the training set into a series of continuous image frames, forming an image data sequence, and inputting the image data sequence into the obtained CNN-LSTM model for training; and a test set is taken to evaluate the trained CNN-LSTM model, and hyper-parameters are adjusted to obtain an optimal CNN-LSTM model. According to the invention, a deep learning network suitable for super-wide dynamic processing of night vision video images is constructed based on a deep learning method, so that the night vision imaging effect under the extremely weak light condition is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video image processing, and in particular relates to a method for processing a video image with super wide dynamic range. Background Art

[0002] Super wide dynamic processing of video images is a technology used to improve the dynamic range of images, which refers to the difference between the brightest and darkest parts of an image. In night vision video image processing, super wide dynamic processing of video images is particularly important because the light conditions at night are often high-contrast, with bright lights and dark shadows. Currently, commonly used wide dynamic processing methods include digital gain control, local tone mapping, multi-frame synthesis, and time domain filtering. However, in night vision video image processing, the digital gain control method may introduce noise, especially in dark areas; the local tone mapping process is complex and computationally intensive, which may lead to a decrease in real-time performance; multi-frame synthesis does not work well for moving objects and may produce motion blur; and time domain filtering may not be effective enough for rapidly changing scenes.

[0003] With the development of artificial intelligence technology, deep learning-based methods have been widely used in video image processing, such as removing noise from videos to improve image quality, adjusting the color of videos to make them more natural or in line with a specific style, etc. However, there are few reports on the use of deep learning methods for wide dynamic range processing of night vision video images. At the same time, due to the complex network structure of deep learning, the amount of calculation may be large, which is difficult to meet the needs of real-time video processing. Summary of the invention

[0004] In view of the above shortcomings, the present invention discloses a method for super wide dynamic processing of video images. Based on the deep learning method, a deep learning network suitable for super wide dynamic processing of night vision video images is constructed and optimized, thereby improving the night vision imaging effect under extremely weak light conditions.

[0005] The present invention is achieved by adopting the following technical solutions:

[0006] A method for processing a video image with super wide dynamic range comprises the following steps:

[0007] (1) constructing a CNN-LSTM model consisting of a CNN part and an LSTM part, wherein the CNN part is used to extract the spatial features of each frame, and the CNN part includes several convolutional layers, pooling layers and activation functions;

[0008] The convolutional layer is expressed as: where Y ij is the element of the output feature map, X m,n is the element of the input image, W is the weight matrix, and b is the bias term;

[0009] Input the features extracted by the CNN part into the LSTM part to extract the dynamic information of the features between frames; the LSTM part includes a forget gate, an input gate and an output gate;

[0010] The input gate is represented as:

[0011] i t =σ(W i ·[h t-1 , x t ]+b i )

[0012]

[0013] The forget gate is expressed as: t =σ(W f ·[h t-1 , x t ]+b f );

[0014] The output gate is represented as:

[0015] Where σ is the Sigmoid function, tanh is the hyperbolic tangent function, and o represents Hadamard multiplication;

[0016] The output of the LSTM part is sent to the fully connected layer for processing to generate a video image after wide dynamic processing;

[0017] (2) collecting video image data under different exposure settings and dividing them into a training set and a test set; decomposing the image data in the training set into a series of continuous image frames and forming an image data sequence, and then inputting the image data sequence into the CNN-LSTM model obtained in step (1) for training, and using the back propagation algorithm to update the model parameters;

[0018] (3) taking the test set to evaluate the CNN-LSTM model trained in step (2) and adjusting the hyperparameters to obtain the optimal CNN-LSTM model, wherein the dynamic range expansion factor and the structural similarity index are used for evaluation;

[0019] (4) Apply the optimal CNN-LSTM model obtained in step (3) to perform super wide dynamic processing on the night vision image data collected by the CIS sensor, and then obtain a high-quality night vision video image after noise reduction and high-quality image reconstruction steps.

[0020] Furthermore, in step (1), the activation function is any one of Leaky ReLU and Leaky ELU;

[0021] The formula of the Leaky ReLU is: Leaky ReLU(x)=max(0.01x,x);

[0022] The formula of the Leaky ELU is: Leaky ELU(x)=max(0,x)+min(0,x)×α×eαx.

[0023] Furthermore, in step (2), the image data sequence is input into the CNN-LSTM model obtained in step (1) for training, wherein the cross entropy loss or mean square error is used as the loss function.

[0024] Furthermore, in step (2), the video image data collected under different exposure settings are first normalized using Z-Score, and then divided into a training set and a test set, wherein the ratio of the training set to the test set is 7:3;

[0025] The formula of the Z-Score is: normalized_value=(value-mean) / std, wherein value is the original pixel value, mean is the average value of the pixel values, and std is the standard deviation of the pixel values.

[0026] Furthermore, in step (2), the image data in the training set is decomposed into a series of continuous image frames and constitutes an image data sequence, and the sequence length is between 5 frames and 20 frames.

[0027] Furthermore, in step (2), during the training of the CNN-LSTM model, a momentum SGD optimizer is used for optimization, and the momentum SGD optimizer rule is expressed as:

[0028] where v t is the momentum vector, β is the momentum coefficient and is set to 0.88.

[0029] Furthermore, in step (4), the optimal CNN-LSTM model obtained in step (3) is applied to perform super wide dynamic processing on the night vision image data collected by the CIS sensor, and then the obtained video image is sharpened and color corrected, and then subjected to noise reduction and high-quality image reconstruction steps to obtain a high-quality night vision video image.

[0030] Compared with the prior art, this technical solution has the following beneficial effects:

[0031] 1. The present invention combines CNN and LSTM to construct a CNN-LSTM model, and uses it for ultra-wide dynamic processing of night vision images, merging multiple frames with different exposure times into an image with a wide dynamic range, solving the problem that traditional cameras cannot capture the details of dark and bright parts at the same time under extremely weak lighting conditions, resulting in overexposure or dark images, thereby improving night vision imaging effects. Among them, CNN can extract rich spatial features from a single frame image, and LSTM can capture the temporal dependency in the image sequence. The combined application of CNN and LTSM enables the model to simultaneously learn and utilize spatial and temporal information, thereby obtaining more comprehensive image features during exposure fusion. Moreover, in exposure fusion, it is usually necessary to process multiple image frames with different exposure settings. The model of the present invention can effectively integrate these multi-frame information, and memorize and utilize the information of the previous frame through the network structure of LSTM, thereby generating a fused image. At the same time, by combining the information of multiple frames of images, the CNN-LSTM model can help expand the dynamic range of the image, so that the fused image can retain more details in the highlight and shadow areas.

[0032] 2. The present invention first uses Z-Score to normalize the video image data collected under different exposure settings, which can accelerate the convergence speed of the network and find the optimal solution more quickly. In addition, by reducing the scale difference between features, Z-Score normalization helps the model to better generalize to new data, especially when the distribution of training data and test data is similar. At the same time, Z-Score normalization is beneficial in avoiding unstable model performance due to changes in the distribution of input data during training. DETAILED DESCRIPTION

[0033] The present invention is further described by the following examples, but is not intended to be limiting of the present invention. Specific experimental conditions and methods not specified in the following examples are conventional methods commonly known to those skilled in the art.

[0034] Embodiment 1: A method for processing super wide dynamic range of a video image, comprising the following steps:

[0035] (1) constructing a CNN-LSTM model consisting of a CNN part and an LSTM part, wherein the CNN part is used to extract the spatial features of each frame, and the CNN part includes several convolutional layers, pooling layers and activation functions;

[0036] The convolutional layer is expressed as: where Y ij is the element of the output feature map, X m,n is the element of the input image, W is the weight matrix, and b is the bias term;

[0037] Input the features extracted by the CNN part into the LSTM part to extract the dynamic information of the features between frames; the LSTM part includes a forget gate, an input gate and an output gate;

[0038] The input gate is represented as:

[0039] i t =σ(W i ·[h t-1 , x t ]+b i )

[0040]

[0041] The forget gate is expressed as: t =σ(W f ·[h t-1 , x t ]+b f );

[0042] The output gate is represented as:

[0043] Where σ is the Sigmoid function, tanh is the hyperbolic tangent function, and o represents Hadamard multiplication;

[0044] The output of the LSTM part is sent to the fully connected layer for processing to generate a video image after wide dynamic processing;

[0045] The activation function is Leaky ReLU; the formula of Leaky ReLU is: Leaky ReLU(x)=max(0.01x,x);

[0046] (2) collecting video image data under different exposure settings, decomposing them into a series of continuous image frames and forming an image data sequence, wherein the sequence length is between 5 and 20 frames; then normalizing them using Z-Score, and then dividing them into a training set and a test set, wherein the ratio of the training set to the test set is 7:3;

[0047] The formula of the Z-Score is: normalized_value=(value-mean) / std, where value is the original pixel value, mean is the average pixel value, and std is the standard deviation of the pixel value;

[0048] Decomposing the image data in the training set into a series of continuous image frames and forming an image data sequence, and then inputting the image data sequence into the CNN-LSTM model obtained in step (1) for training, wherein the cross entropy loss is used as the loss function, and the back propagation algorithm is used to update the model parameters;

[0049] During the training process of the CNN-LSTM model, the momentum SGD optimizer is used for optimization. The momentum SGD optimizer rule is expressed as:

[0050] where v t is the momentum vector, β is the momentum coefficient and is set to 0.88

[0051] (3) taking the test set to evaluate the CNN-LSTM model trained in step (2) and adjusting the hyperparameters to obtain the optimal CNN-LSTM model, wherein the dynamic range expansion factor and the structural similarity index are used for evaluation;

[0052] (4) Apply the optimal CNN-LSTM model obtained in step (3) to perform super wide dynamic processing on the night vision image data collected by the CIS sensor, then sharpen and color correct the obtained video image, and then obtain a high-quality night vision video image after noise reduction and high-quality image reconstruction steps.

[0053] The signal-to-noise ratio of the night vision video image processed by the method described in this embodiment can reach more than 30dB.

[0054] Embodiment 2: A method for processing super wide dynamic range of a video image, comprising the following steps:

[0055] (1) constructing a CNN-LSTM model consisting of a CNN part and an LSTM part, wherein the CNN part is used to extract the spatial features of each frame, and the CNN part includes several convolutional layers, pooling layers and activation functions;

[0056] The convolutional layer is expressed as: where Y ij is the element of the output feature map, X m,n is the element of the input image, W is the weight matrix, and b is the bias term;

[0057] Input the features extracted by the CNN part into the LSTM part to extract the dynamic information of the features between frames; the LSTM part includes a forget gate, an input gate and an output gate;

[0058] The input gate is represented as:

[0059] i t =σ(W i ·[h t-1 , x t ]+b i )

[0060]

[0061] The forget gate is expressed as: t =σ(W f ·[h t-1 , x t ]+b f );

[0062] The output gate is represented as:

[0063] Where σ is the Sigmoid function, tanh is the hyperbolic tangent function, and o represents Hadamard multiplication;

[0064] The output of the LSTM part is sent to the fully connected layer for processing to generate a video image after wide dynamic processing;

[0065] The activation function is Leaky ELU; the formula of Leaky ELU is: Leaky ELU(x)=max(0,x)+min(0,x)×α×e αx ;

[0066] (2) collecting video image data under different exposure settings, decomposing them into a series of continuous image frames and forming an image data sequence, wherein the sequence length is between 5 and 20 frames; then normalizing them using Z-Score, and then dividing them into a training set and a test set, wherein the ratio of the training set to the test set is 7:3;

[0067] The formula of the Z-Score is: normalized_value=(value-mean) / std, where value is the original pixel value, mean is the average pixel value, and std is the standard deviation of the pixel value;

[0068] Decomposing the image data in the training set into a series of continuous image frames and forming an image data sequence, and then inputting the image data sequence into the CNN-LSTM model obtained in step (1) for training, wherein the mean square error is used as the loss function, and the back propagation algorithm is used to update the model parameters;

[0069] During the training process of the CNN-LSTM model, the momentum SGD optimizer is used for optimization. The momentum SGD optimizer rule is expressed as:

[0070] where v t is the momentum vector, β is the momentum coefficient and is set to 0.88

[0071] (3) taking the test set to evaluate the CNN-LSTM model trained in step (2) and adjusting the hyperparameters to obtain the optimal CNN-LSTM model, wherein the dynamic range expansion factor and the structural similarity index are used for evaluation;

[0072] (4) Apply the optimal CNN-LSTM model obtained in step (3) to perform super wide dynamic processing on the night vision image data collected by the CIS sensor, then sharpen and color correct the obtained video image, and then obtain a high-quality night vision video image after noise reduction and high-quality image reconstruction steps.

[0073] The signal-to-noise ratio of the night vision video image processed by the method described in this embodiment can reach more than 30dB.

[0074] Embodiment 3: A method for processing super wide dynamic range of a video image, comprising the following steps:

[0075] (1) constructing a CNN-LSTM model consisting of a CNN part and an LSTM part, wherein the CNN part is used to extract the spatial features of each frame, and the CNN part includes several convolutional layers, pooling layers and activation functions;

[0076] The convolutional layer is expressed as: where Y ij is the element of the output feature map, X m,n is the element of the input image, W is the weight matrix, and b is the bias term;

[0077] Input the features extracted by the CNN part into the LSTM part to extract the dynamic information of the features between frames; the LSTM part includes a forget gate, an input gate and an output gate;

[0078] The input gate is represented as:

[0079] i t =σ(W i ·[h t-1 , x t ]+b i )

[0080]

[0081] The forget gate is expressed as: t =σ(W f ·[h t-1 , x t ]+b f );

[0082] The output gate is represented as:

[0083] Where σ is the Sigmoid function, tanh is the hyperbolic tangent function, and o represents Hadamard multiplication;

[0084] The output of the LSTM part is sent to the fully connected layer for processing to generate a video image after wide dynamic processing;

[0085] The activation function is Leaky ReLU; the formula of Leaky ReLU is: Leaky ReLU(x)=max(0.01x,x);

[0086] (2) collecting video image data under different exposure settings, decomposing them into a series of continuous image frames and forming an image data sequence, wherein the sequence length is between 5 and 20 frames; then normalizing them using Z-Score, and then dividing them into a training set and a test set, wherein the ratio of the training set to the test set is 7:3;

[0087] The formula of the Z-Score is: normalized_value=(value-mean) / std, where value is the original pixel value, mean is the average pixel value, and std is the standard deviation of the pixel value;

[0088] Decomposing the image data in the training set into a series of continuous image frames and forming an image data sequence, and then inputting the image data sequence into the CNN-LSTM model obtained in step (1) for training, wherein the mean square error is used as the loss function, and the back propagation algorithm is used to update the model parameters;

[0089] During the training process of the CNN-LSTM model, the momentum SGD optimizer is used for optimization. The momentum SGD optimizer rule is expressed as:

[0090] where v t is the momentum vector, β is the momentum coefficient and is set to 0.88

[0091] (3) taking the test set to evaluate the CNN-LSTM model trained in step (2) and adjusting the hyperparameters to obtain the optimal CNN-LSTM model, wherein the dynamic range expansion factor and the structural similarity index are used for evaluation;

[0092] (4) Apply the optimal CNN-LSTM model obtained in step (3) to perform super wide dynamic processing on the night vision image data collected by the CIS sensor, then sharpen and color correct the obtained video image, and then obtain a high-quality night vision video image after noise reduction and high-quality image reconstruction steps.

[0093] The signal-to-noise ratio of the night vision video image processed by the method described in this embodiment can reach more than 30dB.

[0094] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for processing super wide dynamic range of video images, characterized in that: The following steps are involved: (1) constructing a CNN-LSTM model consisting of a CNN part and an LSTM part, wherein the CNN part is used to extract the spatial features of each frame, and the CNN part includes several convolutional layers, pooling layers and activation functions; The convolutional layer is expressed as: where Y ij is the element of the output feature map, X m,n is the element of the input image, W is the weight matrix, and b is the bias term; Input the features extracted by the CNN part into the LSTM part to extract the dynamic information of the features between frames; the LSTM part includes a forget gate, an input gate and an output gate; The input gate is represented as: i t =σ(W i ·[h t-1 ,x t ]+b i ) The forget gate is expressed as: t =σ(W f ·[h t-1 , x t ]+b f ); The output gate is represented as: the t =σ(W o ·[h t-1 ,x t ]+b o ) Where σ is the Sigmoid function, tanh is the hyperbolic tangent function, and o represents Hadamard multiplication; The output of the LSTM part is sent to the fully connected layer for processing to generate a video image after wide dynamic processing; (2) Collect video image data under different exposure settings and divide them into training sets and test sets; Decompose the image data in the training set into a series of continuous image frames and form an image data sequence, then input the image data sequence into the CNN-LSTM model obtained in step (1) for training, and use the back propagation algorithm to update the model parameters; (3) taking the test set to evaluate the CNN-LSTM model trained in step (2) and adjusting the hyperparameters to obtain the optimal CNN-LSTM model, wherein the dynamic range expansion factor and the structural similarity index are used for evaluation; (4) Apply the optimal CNN-LSTM model obtained in step (3) to perform super wide dynamic processing on the night vision image data collected by the CIS sensor, and then obtain a high-quality night vision video image after noise reduction and high-quality image reconstruction steps.

2. The method for processing super wide dynamic range of video images according to claim 1, characterized in that: In step (1), the activation function is any one of Leaky ReLU and Leaky ELU; The formula of the Leaky ReLU is: Leaky ReLU(x)=max(0.01x,x); The formula of the Leaky ELU is: Leaky ELU(x)=max(0,x)+min(0,x)×α×e αx .

3. The method for processing super wide dynamic range of video images according to claim 1, characterized in that: In step (2), the image data sequence is input into the CNN-LSTM model obtained in step (1) for training, wherein the cross entropy loss or mean square error is used as the loss function.

4. The method for processing super wide dynamic range of video images according to claim 1, characterized in that: In step (2), the video image data collected under different exposure settings are first normalized using Z-Score, and then divided into a training set and a test set, wherein the ratio of the training set to the test set is 7:3; The formula of the Z-Score is: normalized_value=(value-mean) / std, wherein value is the original pixel value, mean is the average value of the pixel values, and std is the standard deviation of the pixel values.

5. The method for processing super wide dynamic range of video images according to claim 1, characterized in that: In step (2), the image data in the training set is decomposed into a series of continuous image frames and constitutes an image data sequence, and the sequence length is between 5 frames and 20 frames.

6. The method for processing super wide dynamic range of video images according to claim 1, characterized in that: In step (2), during the CNN-LSTM model training process, the momentum SGD optimizer is used for optimization. The momentum SGD optimizer rule is expressed as: where v t is the momentum vector, β is the momentum coefficient and is set to 0.

88.

7. The method for processing super wide dynamic range of video images according to claim 1, characterized in that: In step (4), the optimal CNN-LSTM model obtained in step (3) is applied to perform super wide dynamic processing on the night vision image data collected by the CIS sensor, and then the obtained video image is sharpened and color corrected, and then a high-quality night vision video image is obtained after noise reduction and high-quality image reconstruction steps.

Images