Video noise identification method, device, equipment and computer-readable storage medium

Abstract

The embodiment of the present invention discloses a method for identifying video noise, including: sampling multiple groups of continuous video frames from a video to be identified, respectively calculating the frame difference map corresponding to each group of continuous video frames; inputting the calculated frame difference map into a deep neural network for noise prediction, and outputting a video classification result; wherein the deep neural network is a neural network trained by using the frame difference map of the video as the input of the training neural network; different classification results output by the deep neural network correspond to different video noise severity levels; and outputting a noise identification result according to the video classification result. The present invention is adopted to adaptively learn the characteristics of noise that are different from other texture-intensive areas through the deep neural network, thereby eliminating the interference of texture-intensive areas, thereby greatly improving the accuracy of noise video severity classification.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a video noise recognition method, a video noise recognition device, a video noise recognition equipment and a computer-readable storage medium. Background Art

[0002] In recent years, short video applications have gradually emerged and grown. Short video platforms in existing technologies receive tens of thousands of video updates from users every day, but many of these videos suffer from severe video noise due to the user's shooting light, equipment, etc., which will seriously affect the viewing experience of other users.

[0003] The existing technology for identifying or detecting video noise is mainly divided into two steps. First, the video frame is extracted and the noise of the video frame is analyzed by manually extracting features, and then all the extracted frames are integrated to estimate the noise level of the video. This method of estimating the noise intensity of the video frame using only video frame information and then using the image noise detection algorithm to estimate is often based on the assumption that "noise is high-frequency information in the frequency domain" or "noise areas have high variance in the spatial domain". However, based on observations of a large amount of video data, it is found that texture-intensive video frames, such as video frames of lawns, asphalt roads, oil and salt particles on food, also have the above-assumed characteristics. Therefore, the detection technology in the existing technology cannot accurately distinguish whether texture-intensive video frames have noise problems, and the accuracy is greatly reduced. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a video noise recognition method, a video noise recognition device, a video noise recognition equipment and a computer-readable storage medium, which can distinguish between noise and dense textures and solve the technical problem of low accuracy of detection technology in the prior art.

[0005] In order to solve the above technical problems, an embodiment of the present invention discloses a method for identifying video noise, comprising:

[0006] Sampling multiple groups of continuous video frames from the video to be identified, and calculating the frame difference map corresponding to each group of continuous video frames;

[0007] Inputting the calculated frame difference image into a deep neural network for noise prediction, and outputting a video classification result; wherein the deep neural network is a neural network trained by using the frame difference image of the video as an input for training the neural network; different classification results output by the deep neural network correspond to different video noise severity levels;

[0008] Outputting a noise point recognition result according to the video classification result.

[0009] In conjunction with this aspect, in one possible implementation manner, in the process of respectively calculating the frame difference map corresponding to each group of continuous video frames, for a group of continuous video frames, the process includes:

[0010] Determining a middle video frame in the set of consecutive video frames, and calculating an average frame of the set of consecutive video frames;

[0011] A difference operation is performed between the intermediate video frame and the average video frame to obtain a frame difference map corresponding to the group of continuous video frames.

[0012] In conjunction with this aspect, in one possible implementation manner, performing a difference operation on the intermediate video frame and the average video frame to obtain a frame difference map corresponding to the group of continuous video frames includes:

[0013] After subtracting the average video frame from the intermediate video frame, truncation is performed;

[0014] The frame difference pixel values ​​after truncation processing are translated to the target pixel interval to obtain a frame difference map corresponding to the set of continuous video frames.

[0015] In conjunction with this aspect, in one possible implementation manner, performing a difference operation on the intermediate video frame and the average video frame to obtain a frame difference map corresponding to the group of continuous video frames includes:

[0016] The average video frame is subtracted from the intermediate video frame and an absolute value is taken to obtain a frame difference map corresponding to the set of continuous video frames.

[0017] In conjunction with this aspect, in one possible implementation manner, the group of continuous video frames includes n frames; the truncation process is performed after subtracting the average video frame from the intermediate video frame; and the frame difference pixel value after the truncation process is translated to a target pixel interval to obtain a frame difference map corresponding to the group of continuous video frames, including:

[0018] By formula Calculate and obtain a frame difference map corresponding to the set of continuous video frames;

[0019] Among them, f i,0 is the middle video frame, μ i is the average video frame, κ is the cutoff threshold, 1≤i≤n, and n is a positive integer.

[0020] In conjunction with this aspect, in one possible implementation manner, inputting the calculated frame difference map into a deep neural network for noise prediction includes:

[0021] Shrinking the frame difference image according to a predetermined ratio, wherein the number of pixels of the first side length of the frame difference image after shrinkage is within a first range of values, and the number of pixels of the second side length of the frame difference image after shrinkage is within a second range of values;

[0022] The shrunken frame difference map is input into the deep neural network for noise prediction.

[0023] In conjunction with this aspect, in one possible implementation, the number of pixels of the first side length is 400; the number of pixels of the second side length is 280.

[0024] In conjunction with this aspect, in one possible implementation manner, the training using the frame difference map of the video as an input for training the neural network includes:

[0025] Each video in the video training set is sampled to obtain multiple groups of continuous video frames, and the frame difference map corresponding to each group of continuous video frames is calculated respectively;

[0026] Inputting the calculated frame difference image and the artificial label corresponding to the video into a pre-trained neural network for training; the artificial label is used to indicate the severity of noise in the video;

[0027] The video validation set is used to determine the neural network of the trained prediction model.

[0028] In conjunction with this aspect, in one possible implementation manner, after determining the trained neural network of the prediction model through the video verification set, the method further includes:

[0029] The noise point of the video test set is predicted by the neural network of the determined prediction model, and a confusion matrix of the video classification is generated according to the prediction results and the manual labels corresponding to the video test set.

[0030] In conjunction with this aspect, in one possible implementation, the deep neural network includes a deep neural network constructed by deep separable convolution; and the output of the deep neural network is 3 video classification results.

[0031] Another aspect of the present invention discloses a video noise recognition device, including:

[0032] A sampling calculation unit, used for sampling multiple groups of continuous video frames from the video to be identified, and respectively calculating the frame difference map corresponding to each group of continuous video frames;

[0033] A prediction unit, configured to input the calculated frame difference map into a deep neural network for noise prediction, and output a video classification result; wherein the deep neural network is a neural network trained by using the frame difference map of the video as an input for training the neural network; different classification results output by the deep neural network correspond to different video noise severity levels;

[0034] The recognition result output unit is used to output the noise recognition result according to the video classification result.

[0035] On the other hand, an embodiment of the present invention discloses a video noise recognition device, including a processor and a memory, which are connected to each other, wherein the memory is used to store data processing code, and the processor is configured to call the program code to execute the above-mentioned video noise recognition method.

[0036] Another aspect of the embodiment of the present invention discloses a computer-readable storage medium, wherein the computer storage medium stores program instructions, and when the program instructions are executed by a processor, the processor executes the above-mentioned video noise recognition method.

[0037] The embodiment of the present invention is implemented by sampling multiple groups of continuous frames from the video to be identified, respectively calculating the frame difference map corresponding to each group of continuous frames, inputting the calculated frame difference map into a deep neural network for noise prediction, and outputting a video classification result, wherein the deep neural network is a neural network trained by using the frame difference map of the video as the input of the training neural network; thus, the time information of different frames of the video is utilized, and considering the floating phenomenon of noise in the noisy video, the residual map of the video frame difference is used as the input of the deep neural network, and the deep neural network adaptively learns the characteristics of noise that are different from other texture-intensive areas, which can better distinguish whether the residual pixels in the frame difference information are noise or displacement information of dense texture, and eliminates the interference of texture-intensive areas, thereby greatly improving the accuracy of the classification of the severity of the noisy video. In addition, the method of using video frame difference can greatly reduce the learning difficulty of the neural network, making the training of the neural network easier to converge. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 is an overall architecture diagram of a video noise recognition method provided by an embodiment of the present invention;

[0040] Figure 2 is a flow chart of a method for identifying video noise provided by an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of a process of deep neural network training provided by an embodiment of the present invention;

[0042] Figure 4 is a basic structural diagram of a lightweight neural network provided by an embodiment of the present invention;

[0043] Figure 5 It is an improved network structure diagram provided by the present invention;

[0044] Figure 6 Schematic diagram of an application scenario of the video noise recognition method provided by an embodiment of the present invention;

[0045] Figure 7 is a schematic diagram of an application scenario of a video noise recognition method according to another embodiment of the present invention;

[0046] Figure 8 is a structural schematic diagram of a video noise recognition device provided by an embodiment of the present invention;

[0047] Fig. 9 is a structural schematic diagram of another embodiment of the video noise identification device provided by the present invention;

[0048] Fig.10 It is a structural schematic diagram of a video noise recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In order to better understand the video noise recognition method, device and apparatus provided by the embodiments of the present invention, the overall architecture of the video noise recognition of the embodiments of the present invention is first described below. Figure 1The overall architecture diagram of the video noise recognition method provided by the embodiment of the present invention is shown. For the pre-trained deep neural network, it is first trained through the training video in the video library. Specifically, the frame difference is extracted from the training video, and then the frame difference map and the manual label are used as the input of the deep neural network learning training to obtain the trained deep neural network (a deep neural network for predicting noise or noise points). Finally, the noise points of the video to be predicted (the video to be identified) are predicted by the trained deep neural network. Specifically, the frame difference is extracted from the predicted video (the frame difference extraction method is consistent with that during training), and then the frame difference map is input into the trained deep neural network. Finally, the video classification result is obtained. According to the video classification result, the noise severity of the video to be identified can be known, thereby outputting the noise recognition result.

[0051] The apparatus or device for executing the video noise identification method in the embodiment of the present invention may include but is not limited to network devices such as servers, terminal devices such as desktop computers, laptop computers, tablet computers, and smart terminals. The server may be an independent server or a cluster server. The embodiment of the present invention does not impose any restrictions.

[0052] Let's combine Figure 2 The flowchart of the video noise recognition method provided by the embodiment of the present invention is shown, and the embodiment of the present invention is specifically described. Figure 1 How to use the trained deep neural network to predict the noise of the video to be predicted (the video to be identified) can include the following steps:

[0053] Step S200: sampling multiple groups of continuous video frames from the video to be identified, and calculating the frame difference map corresponding to each group of continuous video frames;

[0054] Specifically, multiple groups of continuous video frames can be randomly sampled from the video to be identified, or multiple groups of continuous video frames can be sampled according to a preset rule (for example, multiple groups of continuous video frames are sampled evenly distributed according to the size of the entire video frame). In one embodiment, in view of the application environment of short videos, the video background in the short video changes little, and after a large amount of data experiments, it is concluded that 10 groups of continuous video frames can be sampled to calculate the frame difference maps corresponding to the 10 groups of continuous video frames respectively.

[0055] In the process of respectively calculating the frame difference map corresponding to each group of continuous video frames, for a group of continuous video frames, the process includes: determining the middle video frame in the group of continuous video frames, and calculating the average frame of the group of continuous video frames; performing a difference operation between the middle video frame and the average video frame to obtain the frame difference map corresponding to the group of continuous video frames.

[0056] Specifically, the intermediate video frame can be subtracted from the average video frame and then truncated, and the frame difference pixel value after truncating can be translated to the target pixel interval, such as the (0, 255) pixel interval or the (0, 101) pixel interval, etc., so as to obtain a frame difference map corresponding to a set of continuous video frames. Alternatively, the intermediate video frame can be subtracted from the average video frame and then the absolute value can be taken without truncation to obtain a frame difference map corresponding to a set of continuous video frames.

[0057] The following is an example of subtracting the average video frame from the intermediate video frame and performing truncation processing, and translating the frame difference pixel values ​​after truncation processing to the target pixel interval to obtain a frame difference map corresponding to a set of continuous video frames:

[0058] The group of continuous video frames may include n frames; in one implementation manner, performing a difference operation between the intermediate video frame and the average video frame to obtain a frame difference map corresponding to the group of continuous video frames may include:

[0059] By formula 1

[0060]

[0061] The frame difference map corresponding to the set of continuous video frames is calculated; wherein, f i,0 is the middle video frame, -,+ represent the video frames before and after the middle video frame, μ i is the average video frame, κ is the cutoff threshold, 1≤i≤n, and n is a positive integer.

[0062] For example, if n is 3, then f i,-1 That is the first frame of the three video frames, f i,0 That is, the second frame of the three video frames, f i,1 That is the third frame of the three video frames.

[0063] For example, if n is 4, then μ i Can be f i,-1 That is the first frame of the three video frames, f i,0 That is the second frame of the three video frames, f i,1 That is the third frame of the three video frames, f i,2 That is, the fourth frame of the three video frames. Or μ i Can be f i,-2 That is the first frame of the three video frames, f i,-1 That is the second frame of the three video frames, f i,0 That is the third frame of the three video frames, f i,1That is the fourth frame of the three video frames.

[0064] It can be seen from Formula 1 that the embodiment of the present invention performs truncation processing on the frame difference map. The truncation processing refers to discarding the pixel difference whose frame difference is greater than the truncation threshold and replacing it with the truncation threshold. Considering that although the noise in the video has significant differences between consecutive frames of the video, the difference between the pixel value at the noise position and the surrounding undamaged pixels is generally small and within the set threshold range. The large difference appearing in the frame difference map is often caused by the non-overlapping of the content between the frames due to the position change. Therefore, the embodiment of the present invention truncates it, which can improve the efficiency of frame difference map extraction and the efficiency of noise detection.

[0065] Moreover, Formula 1 indicates that the frame difference image is increased by the truncation threshold κ, that is, the pixel values ​​of the frame difference image are shifted to between (0, 255), thereby reserving space for recording the pixel difference value of (-255, 255) in the pixel value space of (0, 255) (without scaling this difference value).

[0066] The embodiment of the present invention is not limited to calculating the frame difference map corresponding to a group of continuous video frames by the above formula 1. In another implementation manner, truncation and translation may not be performed, and the intermediate video frame and the average video frame may be directly subjected to difference calculation and then the absolute value is taken, as shown in formula 2:

[0067] ξ i =|f i,0 -μ i | Formula 2

[0068] The frame difference map corresponding to the set of continuous video frames is calculated; wherein, f i,0 is the middle video frame, -,+ represent the video frames before and after the middle video frame, μ i is the average video frame, 1≤i≤n, n is a positive integer.

[0069] In one implementation, n can be 3, 4, 5, etc. The truncation threshold κ can be 50, 86, 127, etc. The embodiment of the present invention conducts comparative experiments to test the effects of different frame difference extraction schemes after training, and it can be concluded that the best effect can be obtained by averaging three frames, that is, n is 3; and the best effect is obtained by taking 50 as the truncation threshold κ and performing translation.

[0070] Step S202: inputting the calculated frame difference image into a deep neural network for noise prediction, and outputting a video classification result;

[0071] Specifically, the deep neural network in the embodiment of the present invention may be a neural network trained using a frame difference map of a video as an input for training the neural network; different classification results output by the deep neural network for noise prediction correspond to different degrees of video noise severity.

[0072] Step S204: outputting a noise point recognition result according to the video classification result.

[0073] The embodiment of the present invention is implemented by sampling multiple groups of continuous frames from the video to be identified, respectively calculating the frame difference map corresponding to each group of continuous frames, inputting the calculated frame difference map into a deep neural network for noise prediction, and outputting a video classification result, wherein the deep neural network is a neural network trained by using the frame difference map of the video as the input of the training neural network; thus, the time information of different frames of the video is utilized, and considering the floating phenomenon of noise in the noisy video, the residual map of the video frame difference is used as the input of the deep neural network, and the deep neural network adaptively learns the characteristics of noise that are different from other texture-intensive areas, which can better distinguish whether the residual pixels in the frame difference information are noise or displacement information of dense texture, and eliminates the interference of texture-intensive areas, thereby greatly improving the accuracy of the classification of the severity of the noisy video. In addition, the method of using video frame difference can greatly reduce the learning difficulty of the neural network, making the training of the neural network easier to converge.

[0074] Combine the following Figures 3 to 5 How to train a deep neural network in the embodiment of the present invention is described in detail. Figure 3 The flowchart of the deep neural network training provided by the embodiment of the present invention may include the following steps:

[0075] Step S300: sampling each video in the video training set to obtain multiple groups of continuous video frames, and calculating the frame difference map corresponding to each group of continuous video frames;

[0076] Specifically, the video database may include a video training set, a video verification set, and a video test set. For example, all videos in the video database are divided according to a preset ratio, and videos that account for a first threshold among all videos are video training sets, videos that account for a second threshold are video verification sets, and videos that account for a third threshold are video test sets. The sum of the first threshold, the second threshold, and the third threshold is 100%, for example, the first threshold is 80%, the second threshold is 20%, and the third threshold is 20%, and so on.

[0077] Then the video training set in the video database can be used to train the pre-trained neural network. Specifically, each video in the video training set is sampled to obtain multiple groups of continuous video frames, and then the frame difference map corresponding to each group of continuous video frames is calculated. The implementation process of extracting the frame difference in step S300 can refer to the above Figure 2 The implementation process of step S200 in the embodiment.

[0078] Step S302: inputting the calculated frame difference image and the artificial label corresponding to the video into a pre-trained neural network for training;

[0079] Specifically, the artificial label in the embodiment of the present invention is used to indicate the severity of the noise in the video; each video in the video database can be manually labeled (i.e., artificial label) to mark which of the three video classifications (severe noise, slight noise, or clear) the video belongs to. The embodiment of the present invention inputs the frame difference image obtained in step S300 and the artificial label corresponding to the video of the frame difference image into the pre-trained neural network for training.

[0080] In one of the embodiments, in order to reduce the computational cost of the neural network, the deep neural network (including pre-trained neural network) in the embodiment of the present invention includes a deep neural network constructed by deep separable convolution; for example, a lightweight neural network (MobileNet) can be selected as the network structure, and the classification output can be changed to three classifications (i.e., 3 video classification results), that is, training according to artificial labels (severe noise, slight noise, no noise).

[0081] MobileNet is an efficient model proposed for mobile and embedded devices. It uses depthwise separable convolution to build a lightweight deep neural network. Its basic structure is as follows Figure 4 As shown in the figure. The first step is depth-wise convolution, which has only M 3x3 convolution kernels. M convolution kernels are convolved with M input maps one by one to obtain M maps, which plays the role of extracting features. The second step is point-wise convolution, which is actually the traditional convolution, but all convolution kernels are 1x1, with a total of M*N 1x1, which plays the role of fusing the extracted features.

[0082] In order to improve the accuracy and speed up the training process, the embodiment of the present invention can use a network pre-trained on the image network (ImageNet) for fine-tuning. Given that image quality belongs to the underlying information and high-level semantic information only plays an auxiliary role, in one implementation, the embodiment of the present invention can simplify MobileNet, including changing the MobileNet network model, removing the last two depth separation convolutional layers in the traditional MobileNet network model, and changing the final classification output to 3, corresponding to the three classifications of noise (severe noise, slight noise, clear), and the network structure is as follows Figure 5 shown.

[0083] Depthwise separable convolution consists of two layers: depthwise convolution and pointwise convolution. Use depthwise convolution to convolve each input channel with a single convolution kernel to obtain the depth of the number of input channels, and then apply pointwise convolution, that is, apply a simple 1x1 convolution, to linearly combine the outputs in the depthwise convolution. MobileNets uses batch normalization (Batch Normalization or Batchnorm, BN) and nonlinear activation units (Rectified Linear Unit) for each layer.

[0084] Depth convolution can use one convolution kernel for each channel; the computational complexity of depth convolution can be: DK*DK*M*DF*DFDK*DK*M*DF*DF.

[0085] Depth convolution is very effective compared to standard convolution, however, it only convolves the input channels and does not combine them to generate new features. Therefore, the next layer uses another layer to use 1x1 convolution to calculate a linear combination of the output of the depth convolution to generate new features.

[0086] Then the combination of depthwise convolution plus 1x1 convolution is called depthwise separable convolution, which was first proposed in (Rigid-motion scattering for image classification.).

[0087] The computational complexity of the depthwise separable convolution can be: DK*DK*M*DF*DF+M*N*DF*DFDK*DK*M*DF*DF+M*N*DF*DF, which is the sum of the depthwise convolution and the 1x1 point-by-point convolution.

[0088] By dividing the convolution into filtering and combination processes, the amount of computation is reduced: DK*DK*M*DF*DF+M*N*DF*DFDK*DK*M*DF*DF=1N+1D2KD K*DK*M*DF*DF+M*N*DF*DFDK*DK*M*DF*DF=1N+1DK2. MobileNet uses 3x3 depthwise separable convolution, which reduces the amount of computation by 8 to 9 times compared to standard convolution.

[0089] Step S304: Determine the neural network of the trained prediction model through the video verification set.

[0090] Specifically, in order to prevent overfitting, the best model is determined from the training results through the video verification set and saved as the prediction model. That is to say, for the video verification set, each video is also sampled, and multiple groups of continuous video frames of the video are sampled, and the frame difference map corresponding to each group of continuous video frames is calculated respectively (the implementation process of extracting frame difference can refer to the above Figure 2 In the implementation process of step S200 in the embodiment, the calculated frame difference map and the artificial label corresponding to the video are input into each trained neural network to determine the model with the best effect.

[0091] In one of the embodiments, when the frame difference map processed by the video training set and the manual labels are input into the pre-trained neural network for training, the optimization method can be set to stochastic gradient descent (SGD), the parameters can be learning rate 1e-3, the potential energy can be 0.9, and the batch size can be 128; and it can be set that each time a first number of video frames (for example, 1000) are processed, verification is performed once through the video verification set, the accuracy is calculated based on the verification results and the manual labels, and the model with the highest accuracy is retained.

[0092] In one of the embodiments, step S304 may further include step S306: performing noise prediction on the video test set through the neural network of the determined optimal model, and generating a confusion matrix for video classification based on the prediction results and the manual labels corresponding to the video test set.

[0093] Specifically, after obtaining the trained neural network of the prediction model, video prediction can be performed using a video test set to obtain prediction results, and then the prediction results can be compared with the manual labels corresponding to the video test set to generate a confusion matrix for video classification to determine how well the deep neural network performs video classification.

[0094] In various embodiments of the present invention, inputting the calculated frame difference map into the deep neural network may further include:

[0095] The frame difference image is shrunk according to a predetermined ratio, the number of pixels of the first side length of the shrunk frame difference image is within a first range value, and the number of pixels of the second side length of the shrunk frame difference image is within a second range value; the shrunk frame difference image is input into a deep neural network for noise prediction.

[0096] Specifically, in order to obtain an input of a fixed size and further reduce the amount of calculation, the frame difference map of the video can be shrunk, and the shrinkage can be performed according to the size ratio of the video itself. For example, the frame difference map can be shrunk to a first side length pixel number (i.e., the number of pixels on the long side of the frame difference map) of 400 and a second side length pixel number (i.e., the number of pixels on the short side of the frame difference map) of 280.

[0097] In one of the embodiments, the video noise recognition method of the embodiment of the present invention can also utilize the attention mechanism, combined with the reward of reinforcement learning (whether the final overall classification is correct), so that the deep neural network can autonomously learn the areas where noise exists in the video frame from a large amount of data, and classify the severity according to the size of these noise areas, the size of the noise particles, the position and other information.

[0098] The video noise recognition method provided by the embodiment of the present invention can be applied in a variety of technical scenarios:

[0099] For example, the inspection personnel of the short video platform review the quality of the videos uploaded by users, such as Figure 6 The schematic diagram of the application scenario of the video noise recognition method provided by the embodiment of the present invention is shown. The video noise recognition device side can extract the frame difference map from the video library in the manner of extracting the frame difference map of the present invention and send it to the pre-trained neural network for training to obtain the trained prediction model neural network, and then extract the frame difference map from the video to be predicted and use the prediction model to predict, and output the video classification result. When it is confirmed according to the output video classification result that the video to be identified does not meet the playback requirements, the recognition of the video to be identified fails; when it is confirmed according to the output video classification result that the video to be identified meets the playback requirements, the recognition of the video to be identified passes. In other words, the video noise recognition method of the present invention can objectively judge the severity of video noise. After being deployed online, it can well replace the manual review of noise, and can automatically and quickly know whether the video to be identified has been identified or reviewed, and can display the review results to the user.

[0100] For example, short video platforms recommend videos to users based on the quality of the videos. Figure 7A schematic diagram of an application scenario of a video noise recognition method according to another embodiment of the present invention is shown. The video noise recognition device side can extract a frame difference map from a video library in the manner of extracting a frame difference map according to the present invention and send the frame difference map to a pre-trained neural network for training to obtain a trained neural network of a prediction model. Then, the frame difference map of a video to be predicted is extracted and predicted using the prediction model, and a video classification result is output. The video is sorted according to the video quality according to the video classification result, and the videos can be recommended to users in order of quality from good to bad.

[0101] The embodiment of the present invention is implemented by sampling multiple groups of continuous frames from the video to be identified, respectively calculating the frame difference map corresponding to each group of continuous frames, inputting the calculated frame difference map into a deep neural network for noise prediction, and outputting a video classification result, wherein the deep neural network is a neural network trained by using the frame difference map of the video as the input of the training neural network; thus, the time information of different frames of the video is utilized, and considering the floating phenomenon of noise in the noisy video, the residual map of the video frame difference is used as the input of the deep neural network, and the deep neural network adaptively learns the characteristics of noise that are different from other texture-intensive areas, which can better distinguish whether the residual pixels in the frame difference information are noise or displacement information of dense texture, and eliminates the interference of texture-intensive areas, thereby greatly improving the accuracy of the classification of the severity of the noisy video. In addition, the method of using video frame difference can greatly reduce the learning difficulty of the neural network, making the training of the neural network easier to converge. Through the embodiment of the present invention, the severity of the video noise can be objectively judged, and after deployment on the line, it can well replace the manual review of the noise.

[0102] In order to better implement the above-mentioned solution of the embodiment of the present invention, the present invention also provides a video noise recognition device, such as Figure 8 FIG. 1 is a schematic diagram of the structure of a video noise recognition device provided by an embodiment of the present invention. The video noise recognition device 80 includes: a sampling calculation unit 800, a prediction unit 802 and a recognition result output unit 804, wherein

[0103] The sampling calculation unit 800 is used to sample multiple groups of continuous video frames from the video to be identified, and calculate the frame difference map corresponding to each group of continuous video frames;

[0104] The prediction unit 802 is used to input the calculated frame difference map into a deep neural network for noise prediction, and output a video classification result; wherein the deep neural network is a neural network trained by using the frame difference map of the video as the input of the training neural network; different classification results output by the deep neural network correspond to different video noise severity levels;

[0105] When it is confirmed according to the output video classification result that the video to be identified does not meet the playback requirement, the identification of the video to be identified fails.

[0106] The sampling calculation unit 800 may include: a determination calculation unit and a difference calculation unit, wherein:

[0107] The determination calculation unit is used to determine the middle video frame in the group of continuous video frames and calculate the average frame of the group of continuous video frames;

[0108] The difference calculation unit is used to perform a difference calculation on the intermediate video frame and the average video frame to obtain a frame difference map corresponding to the group of continuous video frames.

[0109] In one implementation manner, the set of continuous video frames includes n frames; the difference calculation unit may specifically:

[0110] By formula Calculate and obtain a frame difference map corresponding to the set of continuous video frames;

[0111] Among them, f i,0 is the middle video frame, μ i is the average video frame, κ is the cutoff threshold, 1≤i≤n, and n is a positive integer.

[0112] In one of the embodiments, the prediction unit 802 can be used to shrink the frame difference image according to a predetermined ratio, so that the number of pixels of the first side length of the frame difference image after shrinkage is within a first range of values, and the number of pixels of the second side length of the frame difference image after shrinkage is within a second range of values; and the shrunken frame difference image is input into a deep neural network for noise prediction.

[0113] In one implementation manner, the number of pixels of the first side length is 400; the number of pixels of the second side length is 280.

[0114] The recognition result output unit 804 is used to output a noise recognition result according to the video classification result.

[0115] like Fig. 9 The schematic diagram of the structure of another embodiment of the video noise recognition device provided by the present invention is shown. The video noise recognition device 80 includes a sampling calculation unit 800, a prediction unit 802 and a recognition result output unit 804, and may also include a training unit 806, which is used to sample each video in the video training set, sample multiple groups of continuous video frames of the video, and calculate the frame difference map corresponding to each group of continuous video frames;

[0116] Inputting the calculated frame difference image and the artificial label corresponding to the video into a pre-trained neural network for training; the artificial label is used to indicate the severity of noise in the video;

[0117] The video validation set is used to determine the neural network of the trained prediction model.

[0118] After the training unit 806 determines the neural network of the trained prediction model through the video verification set, it can also be used to: perform noise prediction on the video test set through the determined neural network of the prediction model, and generate a confusion matrix for video classification based on the prediction results and the manual labels corresponding to the video test set.

[0119] In one embodiment, the deep neural network includes a deep neural network constructed by deep separable convolution; the output of the deep neural network is three video classification results.

[0120] Each unit of the video noise identification device 80 in the embodiment of the present invention is used to execute the above-mentioned method embodiments. Figures 1 to 5 The steps of the video noise identification method in the embodiment are not repeated here.

[0121] In order to better implement the above-mentioned solution of the embodiment of the present invention, the present invention also provides a video noise recognition device, which is described in detail below with reference to the accompanying drawings:

[0122] like Fig.10 The schematic diagram of the structure of the video noise recognition device provided by the embodiment of the present invention is shown. The video noise recognition device 100 may include a processor 101, a display screen 102, a memory 104 and a communication module 105. The processor 101, the display screen 102, the memory 104 and the communication module 105 may be interconnected via a bus 106. The memory 104 may be a high-speed random access memory (RAM) memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 104 includes the flash in the embodiment of the present invention. The memory 104 may optionally be at least one storage system located away from the aforementioned processor 101. The memory 104 is used to store application code, and may include an operating system, a network communication module, a user interface module and a video noise recognition program. The communication module 105 is used to exchange information and data with external devices. The processor 101 is configured to call the program code and perform the following steps:

[0123] Sampling multiple groups of continuous video frames from the video to be identified, and calculating the frame difference map corresponding to each group of continuous video frames;

[0124] Inputting the calculated frame difference image into a deep neural network for noise prediction, and outputting a video classification result; wherein the deep neural network is a neural network trained by using the frame difference image of the video as an input for training the neural network; different classification results output by the deep neural network correspond to different video noise severity levels;

[0125] Outputting a noise point recognition result according to the video classification result.

[0126] The process of respectively calculating the frame difference map corresponding to each group of continuous video frames by the processor 101 may include, for a group of continuous video frames:

[0127] Determining a middle video frame in the set of consecutive video frames, and calculating an average frame of the set of consecutive video frames;

[0128] A difference operation is performed between the intermediate video frame and the average video frame to obtain a frame difference map corresponding to the group of continuous video frames.

[0129] The processor 101 performs a difference operation on the intermediate video frame and the average video frame to obtain a frame difference map corresponding to the group of continuous video frames, which may include:

[0130] After subtracting the average video frame from the intermediate video frame, truncation is performed;

[0131] The frame difference pixel values ​​after truncation processing are translated to the target pixel interval to obtain a frame difference map corresponding to the set of continuous video frames.

[0132] The processor 101 performs a difference operation on the intermediate video frame and the average video frame to obtain a frame difference map corresponding to the group of continuous video frames, which may include:

[0133] The average video frame is subtracted from the intermediate video frame and an absolute value is taken to obtain a frame difference map corresponding to the set of continuous video frames.

[0134] The group of continuous video frames includes n frames, and the processor 101 performs truncation processing after subtracting the average video frame from the intermediate video frame; and translates the frame difference pixel value after the truncation processing to the target pixel interval to obtain the frame difference map corresponding to the group of continuous video frames, which may include:

[0135] By formula Calculate and obtain a frame difference map corresponding to the set of continuous video frames;

[0136] Among them, f i,0 is the middle video frame, μ i is the average video frame, κ is the cutoff threshold, 1≤i≤n, and n is a positive integer.

[0137] The processor 101 inputs the calculated frame difference map into a deep neural network for noise prediction, which may include:

[0138] Shrinking the frame difference image according to a predetermined ratio, wherein the number of pixels of the first side length of the frame difference image after shrinkage is within a first range of values, and the number of pixels of the second side length of the frame difference image after shrinkage is within a second range of values;

[0139] The shrunken frame difference map is input into the deep neural network for noise prediction.

[0140] Among them, the number of pixels of the first side length is 400; the number of pixels of the second side length is 2100.

[0141] The processor 101 uses the frame difference map of the video as the input for training the neural network for training, which may include:

[0142] Each video in the video training set is sampled to obtain multiple groups of continuous video frames, and the frame difference map corresponding to each group of continuous video frames is calculated respectively;

[0143] Inputting the calculated frame difference image and the artificial label corresponding to the video into a pre-trained neural network for training; the artificial label is used to indicate the severity of noise in the video;

[0144] The video validation set is used to determine the neural network of the trained prediction model.

[0145] After the processor 101 determines the neural network of the trained prediction model through the video verification set, it can also execute:

[0146] The noise point of the video test set is predicted by the neural network of the determined prediction model, and a confusion matrix of the video classification is generated according to the prediction results and the manual labels corresponding to the video test set.

[0147] The deep neural network includes a deep neural network constructed by deep separable convolution; the output of the deep neural network is three video classification results.

[0148] It should be noted that the execution steps of the processor 101 in the video noise recognition device in the embodiment of the present invention can refer to the above-mentioned method embodiments. Figures 1 to 5 The specific implementation of the video noise recognition method in the embodiment will not be described in detail here.

[0149] The embodiment of the present invention is implemented by sampling multiple groups of continuous frames from the video to be identified, respectively calculating the frame difference map corresponding to each group of continuous frames, inputting the calculated frame difference map into a deep neural network for noise prediction, and outputting a video classification result, wherein the deep neural network is a neural network trained by using the frame difference map of the video as the input of the training neural network; thus, the time information of different frames of the video is utilized, and considering the floating phenomenon of noise in the noisy video, the residual map of the video frame difference is used as the input of the deep neural network, and the deep neural network adaptively learns the characteristics of noise that are different from other texture-intensive areas, which can better distinguish whether the residual pixels in the frame difference information are noise or displacement information of dense texture, and eliminates the interference of texture-intensive areas, thereby greatly improving the accuracy of the classification of the severity of the noisy video. In addition, the method of using video frame difference can greatly reduce the learning difficulty of the neural network, making the training of the neural network easier to converge. Through the embodiment of the present invention, the severity of the video noise can be objectively judged, and after deployment on the line, it can well replace the manual review of the noise.

[0150] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0151] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A video noise recognition method, It is characterized in that include: Sampling multiple groups of continuous video frames from the video to be identified, and calculating the frame difference map corresponding to each group of continuous video frames respectively; wherein, for a group of continuous video frames, determining the middle video frame in the group of continuous video frames, and calculating the average video frame of the group of continuous video frames; After subtracting the average video frame from the intermediate video frame, truncation is performed; The frame difference pixel values ​​after truncation processing are translated to the target pixel interval to obtain a frame difference image corresponding to the set of continuous video frames; Inputting the calculated frame difference image into a deep neural network for noise prediction, and outputting a video classification result; wherein the deep neural network is a neural network trained by using the frame difference image of the video as an input for training the neural network; different classification results output by the deep neural network correspond to different video noise severity levels; Outputting a noise point recognition result according to the video classification result.

2. The method according to claim 1, It is characterized in that The step of performing a difference operation between the intermediate video frame and the average video frame to obtain a frame difference map corresponding to the group of continuous video frames further includes: The average video frame is subtracted from the intermediate video frame and an absolute value is taken to obtain a frame difference map corresponding to the set of continuous video frames.

3. The method according to claim 1, It is characterized in that The group of continuous video frames includes n frames; the intermediate video frame is subtracted from the average video frame and then truncated; the frame difference pixel value after truncation is translated to the target pixel interval to obtain a frame difference map corresponding to the group of continuous video frames, including: By formula Calculate and obtain a frame difference map corresponding to the set of continuous video frames; in, is the middle video frame, , Indicates that relative to the video frame before the intermediate video frame, the represents a video frame relative to the intermediate video frame, is the average video frame, is the cutoff threshold, , n is a positive integer.

4. The method according to claim 1, It is characterized in that The step of inputting the calculated frame difference image into a deep neural network for noise prediction includes: Shrinking the frame difference image according to a predetermined ratio, wherein the number of pixels of the first side length of the frame difference image after shrinkage is within a first range of values, and the number of pixels of the second side length of the frame difference image after shrinkage is within a second range of values; The shrunken frame difference map is input into the deep neural network for noise prediction.

5. The method according to any one of claims 1 to 4, It is characterized in that The training using the frame difference map of the video as the input of the training neural network includes: Each video in the video training set is sampled to obtain multiple groups of continuous video frames, and the frame difference map corresponding to each group of continuous video frames is calculated respectively; Inputting the calculated frame difference image and the artificial label corresponding to the video into a pre-trained neural network for training; the artificial label is used to indicate the severity of noise in the video; The video validation set is used to determine the neural network of the trained prediction model.

6. The method according to claim 5, It is characterized in that After determining the trained neural network of the prediction model through the video validation set, the method further includes: Performing noise prediction on a video test set by using the determined neural network of the prediction model; A confusion matrix for video classification is generated based on the prediction results and the manual labels corresponding to the video test set.

7. A video noise recognition device, It is characterized in that include: A sampling calculation unit, used for sampling multiple groups of continuous video frames from the video to be identified, and respectively calculating the frame difference map corresponding to each group of continuous video frames; A prediction unit, configured to input the calculated frame difference map into a deep neural network for noise prediction, and output a video classification result; wherein the deep neural network is a neural network trained by using the frame difference map of the video as an input for training the neural network; different classification results output by the deep neural network correspond to different video noise severity levels; A recognition result output unit, used to output a noise recognition result according to the video classification result; Wherein, the sampling calculation unit includes: A determination calculation unit is used to determine a middle video frame in a group of continuous video frames and calculate an average video frame of the group of continuous video frames; The difference operation unit is used to perform a difference operation on the intermediate video frame and the average video frame to obtain a frame difference map corresponding to the group of continuous video frames, including: subtracting the average video frame from the intermediate video frame, performing a truncation process, and translating the frame difference pixel value after the truncation process to a target pixel interval to obtain a frame difference map corresponding to the group of continuous video frames.

8. A video noise recognition device, It is characterized in that The method comprises a processor and a memory, wherein the processor and the memory are connected to each other, wherein the memory is used to store program codes, and the processor is configured to call the program codes to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores program instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images