Multi-frame Fusion Video Noise Evaluation Method Based on Deep Convolutional Neural Network

Abstract

The present invention discloses a multi-frame fusion video noise evaluation method based on a deep convolutional neural network, which mainly solves the problems that existing evaluation methods cannot accurately fit the real scene, have a narrow noise coverage range and low evaluation accuracy for ultra-high definition videos in video noise evaluation. The implementation scheme is as follows: ultra-high definition videos with different noises are collected through two methods, artificial degradation and real shooting, and they are classified into levels to form a training set; a multi-frame fusion video noise evaluation network is designed, and the training set is used to train this network; the video to be evaluated is grouped into video frames according to the frame sequence, and the grouped video frames are input into the trained network to output the noise level of each frame of the video to be evaluated; the level of the frame with the most outputs is used as the level of the video to be evaluated. The present invention has a wide coverage range and can better fit the real scene. It not only has a high evaluation accuracy for the video noise level, but also has a faster evaluation speed. It can be used in ultra-high definition video noise detection systems.

Description

Technical Field

[0001] The present invention belongs to the technical field of image and video processing, and further relates to a method for evaluating multi-frame fusion video noise, which can be used in a system for detecting ultra-high definition video noise. Background Art

[0002] When a video is acquired, due to the interference of sensor material properties, working environment, electronic components and circuit structures, as well as transmission media and recording devices during the transmission process, the generation of noise is inevitable. The existence of noise will disrupt observable information and affect the video quality to a certain extent. In today's video industry, many videos have different noise distortions, and different pushing intensities need to be applied according to the perceived quality of the video noise level by users; for video surveillance, due to the different noise levels, the impact on observable information is very large. If unified noise reduction processing is adopted, some observable information will be lost. Therefore, it is very necessary to perform corresponding noise reduction processing according to different noise levels. Currently, the commonly used method for evaluating video noise is obtained based on Gaussian white noise modeling or mathematical statistical analysis. Although this method can play a certain role in noise evaluation, it is time-consuming and laborious, and there will be problems where the scene cannot be accurately evaluated when applied to real scenarios. When applied to a system for detecting ultra-high definition video noise, it takes a long time and has a low accuracy. Therefore, it is necessary to establish a method for objectively, automatically, and with low time consumption to evaluate the noise level of ultra-high definition videos.

[0003] Northwestern Polytechnical University disclosed a method for evaluating image noise estimation in its patent document with the patent number CN201911066796.X. The implementation steps of this method include: collecting noise-free images of any scene and performing data augmentation on them. Further intercept texture structure regions with a pixel size of 200x200 from the augmented pictures, and divide all training pictures into 50x50 image blocks in an overlapping manner to form a training dataset for training a deep convolutional neural network. During training, randomly generated numbers are used to construct noise-polluted images, and each time a noise-polluted image t i,s The corresponding noise simulation diagram n i,s is used as the input of the neural network, and the corresponding label is n i,s i,s ​During detection, only an arbitrarily noise - contaminated image needs to be input into the trained deep convolutional neural network to obtain the corresponding noise map. Then, histogram statistics are performed on the noise map to estimate the corresponding noise distribution, and the mean square error is calculated to obtain the corresponding noise level. Although this method uses a deep neural network to predict the noise map and calculates the noise level by statistically analyzing the distribution of the noise map, there is a certain improvement in prediction speed and accuracy. However, this method still has three deficiencies:

[0004] First, in the production of its dataset, due to the existence of photon noise during the shooting process, the influence of electronic components and circuit structures, and the interference of the transmission medium during the transmission process, it is impossible to obtain completely noise - free images;

[0005] Second, during training, a method of generating random numbers is used to obtain the corresponding noise simulation map. However, the noise in the real world is difficult to model, and using random numbers to model the noise simulation map has poor accuracy in real scenarios;

[0006] Third, this method mainly targets noise images and has low accuracy in evaluating the noise of ultra - high - definition videos. Summary of the Invention

[0007] The purpose of the present invention is to propose a multi - frame fusion video noise evaluation method based on a deep convolutional neural network to solve the problems in the prior art that it cannot accurately fit the real - world scenario, has a narrow noise coverage range, and has low accuracy in evaluating ultra - high - definition videos in video noise evaluation.

[0008] The technical idea of the present invention is as follows: By collecting ultra - high - definition videos with different noise levels through two methods, artificial degradation and real acquisition, to form a training set, the problem that the video noise evaluation cannot accurately fit the real - world scenario is solved; by classifying the noise videos into levels, the problem of insufficient noise coverage is solved; by designing a multi - frame fusion deep neural classification network, the accuracy of evaluating the noise of ultra - high - definition videos is improved.

[0009] According to the above idea, the implementation steps of the present invention are as follows:

[0010] (1) Generate an ultra - high - definition video training set containing five noise - level categories:

[0011] 1a) Select videos from the Train_91 dataset and degrade them in a Gaussian degradation manner. According to the Gaussian noise degradation range, the videos are divided into five categories: negligible noise pollution, slight noise pollution, obvious noise pollution, severe noise pollution, and extremely severe noise pollution, corresponding to noise levels of 0, 1, 2, 3, and 4 respectively;

[0012] 1b) Obtain various noisy videos in real situations through camera shooting, and classify their noise levels with reference to the degraded video categories in 1a).

[0013] 1c) Combine the degraded videos and real videos with their respective category labels after the above level classification to form a training set.

[0014] (2) Construct a multi-frame fusion video noise evaluation network based on a deep convolutional neural network:

[0015] 2a) Establish a basic feature extraction module composed of 1 input convolutional block, 2 convolutional downsampling blocks, 2 convolutional upsampling pooling blocks, and 1 output convolutional block cascaded in sequence.

[0016] 2b) Parallelly connect 3 basic feature modules, and then cascade them with 1 basic feature extraction module, an adaptive average pooling layer, and a fully connected layer in sequence to form a multi-frame fusion video noise evaluation network.

[0017] (3) Train the multi-frame fusion video noise evaluation network:

[0018] 3a) For each video in the training set, select two groups of five video frames, first crop them to a pixel size of 224*224, and then store them in the lmdb format.

[0019] 3b) Input the training set in the lmdb format into the multi-frame fusion video noise evaluation network and perform iterative training on it using the stochastic gradient descent method to obtain a trained multi-frame fusion video noise evaluation network.

[0020] (4) Evaluate the noise level of the ultra-high-definition video:

[0021] 4a) Group the ultra-high-definition video with the frame to be evaluated as the center, taking two frames before and after it as a group.

[0022] 4b) Input the grouped video frames into the trained multi-frame fusion video noise evaluation network frame by frame in sequence, output the evaluation category of the frame to be evaluated, and count the most frequently occurring evaluation category to obtain the noise level of the ultra-high-definition video.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] First, since the training set generated by the present invention adopts two production methods of artificial degradation and real acquisition, it not only solves the problem of too small data set, but also solves the problem of insufficient fitting of the data set to the real scene, making the present invention cover a wider range of pollution degrees caused by noise, have higher accuracy, and be more in line with the real noise situations that occur in practice when objectively evaluating the video noise quality.

[0025] Second, since the present invention divides the ultra-high-definition video images into five categories with negligible noise pollution, slight noise pollution, obvious noise pollution, serious noise pollution, and extremely serious noise pollution, covering the video noise conditions in various scenarios, the trained neural network has learned the differences between different noise levels, making the results of the present invention in objectively evaluating the video noise quality more extensive and accurate.

[0026] Third, since the present invention uses multi-frame fusion input when extracting noise features and refers to the frame information of the previous and next frames when evaluating the current frame, the network can obtain more information, overcoming the problem of errors in noise evaluation caused by interference in the video, making the prediction accuracy of the network higher; at the same time, a deep neural network is used to improve the evaluation efficiency of ultra-high-definition video images. Brief Description of the Drawings

[0027] Figure 1 It is a flowchart for the implementation of the present invention.

[0028] Figure 2 It is a structural diagram of a multi-frame fusion video noise evaluation network constructed in the present invention. Detailed Embodiment

[0029] The embodiments and effects of the present invention will be described in detail below with reference to the drawings.

[0030] Refer to Figure 1 , the implementation steps of this example are as follows:

[0031] Step 1, establish a training set for ultra-high-definition noise video data.

[0032] The frequency dataset in this step is obtained by two methods: artificial degradation and real acquisition. The specific implementation is as follows:

[0033] 1.1) Obtain the frequency dataset by artificial degradation, and perform category division and calibrate the noise level:

[0034] 1.1.1) Perform video acquisition from the Train_91 dataset, which includes common videos such as people, animals, plants, sports scenes, and static scenes. Use the functions embedded in python to generate a noise matrix that conforms to the Gaussian distribution, add it to the original image, and perform Gaussian noise degradation;

[0035] 1.1.2) For the videos screened from Train_91, divide their noise levels according to the Gaussian noise degradation range. That is, define the degradation of Gaussian noise in the range of 0 - 5 as negligible noise pollution, marked as level 0; define the degradation of Gaussian noise in the range of 5 - 10 as slight noise pollution, marked as level 1; define the degradation of Gaussian noise in the range of 10 - 15 as obvious noise pollution, marked as level 2; define the degradation of Gaussian noise in the range of 15 - 20 as severe noise pollution, marked as level 3; define the degradation of Gaussian noise in the range of 20 - 30 as extremely severe noise pollution, marked as level 4.

[0036] 1.2) Obtain the real video dataset and conduct category division, and calibrate the noise levels:

[0037] 1.2.1) Use a camera to shoot videos in different scenarios such as roads, campuses, scenic spots, and various scenarios of people, animals, movement, and stillness, and obtain an ultra-high-definition real video dataset with different noises by adjusting the ISO and exposure settings of the camera;

[0038] 1.2.2) Organize more than N > 8 annotators to label the levels of the obtained real videos. That is, play the videos using PotPlayer in a well-lit environment and a monitor with a resolution of 1920×1080. The annotators first select a degraded video as a reference from the artificially degraded videos, and then, with this video as the reference standard, each annotator divides and labels the noise levels of the real videos under the unified reference standard, so as to divide the real videos into five levels of negligible noise pollution, slight noise pollution, obvious noise pollution, severe noise pollution, and extremely severe noise pollution, and respectively correspond them to five level labels of level 0, level 1, level 2, level 3, and level 4;

[0039] 1.2.3) Select the results of the majority of people's video level annotations as the final noise levels of the real video data;

[0040] 1.3) Combine the degraded videos with their respective category labels divided in step 1.1) and the real videos with their respective category labels divided in step 1.2) to form a training set.

[0041] Step 2, construct a multi-frame fusion video noise evaluation network based on a deep convolutional neural network.

[0042] Refer to Figure 2 , the specific implementation of this step is as follows:

[0043] 2.1) Construct a basic feature extraction module:

[0044] Select one input convolutional block, which includes two convolutional layers. The convolutional kernel size of each convolutional layer is 3x3, the stride is 1, the padding is 1, and the activation function is Relu; the input and output channels of the first convolutional layer are (9, 90), and the input and output channels of the second convolutional layer are (90, 32).

[0045] Select two convolutional downsampling blocks. Each convolutional downsampling block includes three convolutional layers. The convolutional kernel size of each convolutional layer is 3x3, the stride is 1, and the padding is 1; the input and output channels of the three convolutional layers of the first convolutional downsampling block are (32, 64), (64, 64), and (64, 64) respectively; the input and output channels of the three convolutional layers of the second convolutional downsampling block are (64, 128), (128, 128), and (128, 128) respectively.

[0046] Select two convolutional upsampling pooling blocks. Each convolutional upsampling pooling block includes three convolutional layers and one pixel shuffle upsampling layer. The convolutional kernel size of each convolutional layer is 3x3, the stride is 1, the padding is 1, and the upsampling factor of the pixel shuffle upsampling layer is 2; the input and output channels of the three convolutional layers of the first convolutional downsampling block are (128, 128), (128, 128), and (128, 256) respectively; the input and output channels of the three convolutional layers of the second convolutional downsampling block are (64, 64), (64, 64), and (64, 128) respectively.

[0047] Select one output convolutional block, which includes two convolutional layers. The convolutional kernel size of each convolutional layer is 3x3, the stride is 1, and the padding is 1; the input and output channels of the first convolutional layer are (32, 32), and the input and output channels of the second convolutional layer are (32, 3).

[0048] Cascade the above one input convolutional block, two convolutional downsampling blocks, two convolutional upsampling blocks and one output convolutional block in sequence to form a basic feature extraction module;

[0049] 2.2) Construct a multi-frame fusion video noise evaluation network:

[0050] Select 4 basic feature extraction modules constructed in 2.1);

[0051] Select one adaptive average pooling layer with its parameter set to 1x1;

[0052] Select one fully connected layer with its input channels being 3 and output channels being 5;

[0053] Parallelly connect the above 3 basic feature modules, and then cascade them with 1 basic feature extraction module, the adaptive average pooling layer, and the fully connected layer in sequence to form a multi-frame fusion video noise evaluation network.

[0054] Step 3, train the multi-frame fusion video noise evaluation network.

[0055] 3.1) Configure the training environment and install the required python libraries for network training;

[0056] 3.2) Set the batch size to 16, the initial learning rate to 0.001, the weight decay rule to keep the learning rate unchanged for the first 20 training epochs, and multiply the learning rate by 0.1 after the 20th training epoch. Select SGD as the solver, set the training epoch to 150, and set the loss function of the network to the cross-entropy loss function:

[0057]

[0058] Where L represents the loss value; i represents the index value of the image in each training batch; N represents the batch size, which is set to 16 in this embodiment; j represents the category index value of the noise level classification; M represents the total number of categories of the noise level classification, which is set to 5 in this embodiment; p ij represents the probability that the noise level category of the i-th image output by the network in each training batch is equal to the j-th noise level category; y ij is 0 or 1. If the true noise level category of the i-th image in each training batch is equal to the j-th true noise level category, then y ij is 1, otherwise, y ij is 0; [[ID=2V]]

[0059] 3.3) For each video in the training set, select two groups of five-frame video frames, first crop them to a pixel size of 224*224, and then store them in the lmdb format;

[0060] 3.4) Use the stochastic gradient descent method to train the multi-frame fusion video noise evaluation network:

[0061] 3.4.1) In each training epoch, first input the training set in the lmdb format into the network in batches according to the batch size for forward propagation, and calculate the loss value between the output value and the target value;

[0062] 3.4.2) According to the calculated loss value, use the stochastic gradient descent method SGD to update the network parameters;

[0063] 3.4.3) Repeat 3.4.1) to 3.4.2) until the loss function converges or reaches the set training epoch to obtain the trained multi-frame fusion video noise evaluation network.

[0064] ]>Step 4, evaluate the ultra-high definition video with unknown noise level.

[0065] 4.1) Obtain the noisy ultra-high definition video:

[0066] Obtain artificially degraded noisy ultra-high-definition videos: Obtain 20 ultra-high-definition videos from the Internet. Using the Gaussian degradation method described in step 1.1.1), when performing degradation, randomly set the degradation range to obtain artificially degraded noisy ultra-high-definition videos with unknown noise levels.

[0067] Obtain real-shot noisy ultra-high-definition videos: By setting different parameters of the camera, such as ISO value and exposure time, obtain real-shot noisy ultra-high-definition videos.

[0068] 4.2) Group the two types of noisy ultra-high-definition videos obtained in 4.1) with the frame to be evaluated as the center, taking two frames before and after as a group. Input the grouped video frames into the trained multi-frame fusion video noise evaluation network frame by frame in sequence, and output the evaluation level of the frame to be evaluated.

[0069] 4.3) According to the evaluation level of the frame to be evaluated output by the network, count the level of the frame with the most occurrences of the evaluation frame to obtain the noise level of the ultra-high-definition video, and complete the evaluation of the multi-frame fusion video noise.

[0070] The effects of the present invention can be further illustrated by the following simulation experiments:

[0071] 1. Simulation experiment conditions:

[0072] The hardware platform for the simulation experiment of the present invention is: The CPU is Intel i9 9820x, the memory is 16GB, and the GPU is GTX 1080Ti;

[0073] The software platform for the simulation experiment of the present invention is: Linux operating system and python 3.6;

[0074] The ultra-high-definition noisy videos for the simulation experiment of the present invention are obtained through two methods: artificial degradation and camera shooting. A total of 65 videos are produced and collected, including 40 artificially degraded noisy videos and 30 camera-shot noisy videos. The number of frames in each video ranges from 150 to 500, the frame rate ranges from 25fps to 60fps, and the resolutions include 720p, 1080p, 2K, and 4K. The videos cover most common scenarios; the obtained ultra-high-definition noisy videos are used to test the accuracy of the multi-frame fusion video noise evaluation network based on a deep convolutional neural network constructed by the present invention for evaluating the noise video level.

[0075] 2. Simulation content and its result analysis:

[0076] Simulation 1: Under the above simulation conditions, the method of the present invention is used to construct a test set of ultra-high-definition noisy video data, and the noise level of each video in the constructed test set of noisy video data is evaluated. The average noise level evaluation accuracy rate of the test set is calculated, and the prediction accuracy rates of the multi-frame fusion video noise evaluation network based on the deep convolutional neural network constructed by the present invention on the five noise level categories of the video noise level test set are obtained, as shown in Table 1;

[0077] Table 1 List of prediction accuracy rates of the video noise level test set

[0078] Category Accuracy Level 0: Noise pollution is negligible 0.97 Level 1: Slight noise pollution 0.96 Level 2: Obvious noise pollution 0.94 Level 3: Severe noise pollution 0.95 Level 4: Extremely severe noise pollution 0.94

[0079] As can be seen from Table 1, the multi-frame fusion video noise evaluation network based on the deep convolutional neural network constructed by the present invention has an accuracy rate of 94% or more in the noise level evaluation of the five levels of the ultra-high-definition video test set, and has a relatively high level evaluation accuracy rate.

[0080] Simulation 2: Under the above simulation conditions, the present invention is used to construct a multi-resolution noisy video test set with resolutions of 720p, 1080p, 2K, and 4K, and the time used for each frame of the test set is calculated using the time function of python, and the average single-frame time used on videos with different resolutions is obtained, as shown in Table 2;

[0081] Table 2 List of average single-frame time used by the present invention on videos with different resolutions

[0082] Video resolution Average time per single frame 1280×720(720p) 0.03s 1920×1080(1080p) 0.03s 2560×1440(2K) 0.04s 3840×2160(4K) 0.12s

[0083] As can be seen from Table 2, the multi-frame fusion video noise evaluation network based on the deep convolutional neural network constructed by the present invention takes a short time to evaluate multi-resolution ultra-high-definition videos. The processing time for each frame of the 4K ultra-high-definition video evaluation is only 0.12 seconds, indicating that the present invention can accurately and quickly evaluate the noise level of large-resolution videos;

[0084] The above simulation results show that the present invention can not only ensure the accuracy rate of noise level evaluation but also ensure the processing speed, and both have achieved good evaluation effects.

Claims

1. A multi-frame fusion video noise evaluation method based on a deep convolutional neural network, characterized in that It includes the following steps: (1) Generate an ultra-high-definition video training set containing five noise level categories: 1a) Select videos from the Train_91 dataset, degrade them in the way of Gaussian degradation, and divide the videos into five categories of negligible noise pollution, slight noise pollution, obvious noise pollution, severe noise pollution, and extremely severe noise pollution according to the Gaussian noise degradation range, corresponding to category labels 0, 1, 2, 3, and 4 respectively; 1b) Obtain various noisy videos in real situations by camera shooting, and divide their noise levels with reference to the video categories degraded in 1a); 1c) Combine the degraded videos and real videos with their respective category labels that have been classified into a training set; (2) Construct a multi-frame fusion video noise evaluation network based on a deep convolutional neural network: 2a) Establish a basic feature extraction module composed of 1 input convolutional block, 2 convolutional downsampling blocks, 2 convolutional upsampling pooling blocks, and 1 output convolutional block cascaded in sequence; 2b) Parallelize 3 basic feature modules, and then cascade them with 1 basic feature extraction module, an adaptive average pooling layer, and a fully connected layer in sequence to form a multi-frame fusion video noise evaluation network; (3) Train the multi-frame fusion video noise evaluation network: 3a) For each video in the training set, select two groups of five video frames, first crop them to a pixel size of 224*224, and then store them in the lmdb format; 3b) Input the training set in the lmdb format into the multi-frame fusion video noise evaluation network and use the stochastic gradient descent method to perform iterative training on it to obtain a trained multi-frame fusion video noise evaluation network; (4) Evaluate the noise level of the ultra-high-definition video: 4a) Take two frames before and after the frame to be evaluated as a group for the ultra-high-definition video for grouping; 4b) Input the grouped video frames into the trained multi-frame fusion video noise evaluation network frame by frame sequence by group, output the evaluation category of the frame to be evaluated, and count the most frequently occurring evaluation category to obtain the ultra-high-definition video noise level.

2. The method according to claim 1, characterized in that The input convolutional block in 2a) includes two convolutional layers. The convolutional kernel size of each convolutional layer is 3x3, the stride is 1, and the padding is 1; the input and output channels of its first convolutional layer are (9, 90), and the input and output channels of the second convolutional layer are (90, 32).

3. The method according to claim 1, wherein The structural parameters of the two convolutional downsampling blocks in 2a) are as follows: Both of the two convolutional downsampling blocks include three convolutional layers, and the convolutional kernel size of each convolutional layer is 3x3, the stride is 1, and the padding is 1; The input and output channels of the three convolutional layers of the first convolutional downsampling block are (32, 64), (64, 64), and (64, 64) respectively; The input and output channels of the three convolutional layers of the second convolutional downsampling block are (64, 128), (128, 128), and (128, 128) respectively.

4. The method according to claim 1, characterized in that, The structural parameters of the two convolutional upsampling pooling blocks in 2a) are as follows: Both of the two convolutional upsampling pooling blocks include three convolutional layers and one pixel shuffle upsampling layer. The convolutional kernel size of each convolutional layer is 3x3, the stride is 1, the padding is 1, and the upsampling factor of the pixel shuffle upsampling layer is 2; For the three convolutional layers of the first convolutional downsampling block, the input and output channels are (128, 128), (128, 128), and (128, 256) respectively; For the three convolutional layers of the second convolutional downsampling block, the input and output channels are (64, 64), (64, 64), and (64, 128) respectively.

5. The method according to claim 1, wherein The output convolutional block in 2a) includes two convolutional layers. The convolutional kernel size of each convolutional layer is 3x3, the stride is 1, and the padding is 1; the input and output channels of its first convolutional layer are (32, 32), and the input and output channels of the second convolutional layer are (32, 3).

6. The method according to claim 1, characterized in that, In the training set based on the lmdb format in 3b), the multi-frame fusion video noise evaluation network is iteratively trained using the stochastic gradient descent method as follows: 3b1) Set the batch size to 16, the initial learning rate to 0.001, the weight decay rule to keep the learning rate unchanged for the first 20 training epochs, multiply the learning rate by 0.1 after the 20th training epoch, select SGD as the solver, set the training epoch to 150, and set the loss function of the network to the cross-entropy loss function; 3b2) In each training epoch, first input the training set into the network in batches according to the batch size for forward propagation, and calculate the loss value between the output value and the target value; 3b3) Update the network parameters using the SGD method according to the loss value calculated in 3b2); 3b4) Repeat 3b2) to 3b3) until the loss function converges or reaches the set training epoch to obtain the trained multi-frame fusion video noise evaluation network.

7. The method according to claim 1, characterized in that For the videos selected from Train_91 in 1a), the noise level is divided according to the Gaussian noise degradation range as follows: Define the degradation of Gaussian noise in the range of 0 - 5 as negligible noise pollution, marked as level 0, Define the degradation of Gaussian noise in the range of 5 - 10 as slight noise pollution, marked as level 1, Define the degradation of Gaussian noise in the range of 10 - 15 as obvious noise pollution, marked as level 2, Define the degradation of Gaussian noise in the range of 15 - 20 as severe noise pollution, marked as level 3, Define the degradation of Gaussian noise in the range of 20 - 30 as extremely severe noise pollution, marked as level 4.

8. The method according to claim 1, wherein In 1b), take multiple noisy videos in the real situation and perform noise level annotation with reference to the video categories degraded in 1a) as follows 1b1) Set N annotators to first select the degraded videos as references from the videos artificially degraded in 1a), and then use the reference video as the reference standard to divide the noise level of the video to be annotated, where N is greater than 8; 1b2) Select the result of the majority of people's video level annotation as the final noise level of the video.

Images