Video denoising method and device, equipment, storage medium and program product

Abstract

The invention relates to a video denoising method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining an original video sequence, carrying out the data compression of the original video sequence, obtaining a compressed video sequence, carrying out the optical flow alignment operation of the compressed video sequence according to a preset denoising model, and obtaining an optical flow alignment result; denoising the compressed video sequence after optical flow alignment to obtain a denoised video sequence; the de-noising model is obtained by pruning and quantifying the initial de-noising model. According to the method, the real-time performance of video processing can be improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video denoising method, device, equipment, storage medium and program product. Background Art

[0002] With the development of video technology, the application fields of video are becoming more and more extensive.

[0003] Taking a sports camera as an example, a user can usually use a sports camera to shoot a video of a sports scene. The video shot by the sports camera may be mixed with various noises, so that the video becomes blurred, resulting in a decrease in the quality of the captured video. In the related art, the captured video can be denoised to obtain a high-quality video.

[0004] However, the video denoising method in the related art has a slow denoising speed, resulting in low real-time performance of video processing. Summary of the invention

[0005] Based on this, it is necessary to provide a video denoising method, device, equipment, storage medium and program product to improve the real-time performance of video processing in response to the above technical problems.

[0006] In a first aspect, the present application provides a video denoising method, comprising:

[0007] Get the original video sequence;

[0008] Performing data compression on the original video sequence to obtain a compressed video sequence;

[0009] An optical flow alignment operation is performed on the compressed video sequence according to a preset denoising model, and the compressed video sequence after the optical flow alignment is denoised to obtain a denoised video sequence; the denoising model is obtained by pruning and quantizing the initial denoising model.

[0010] In one embodiment, compressing an original video sequence to obtain a compressed video sequence includes:

[0011] The original video sequence is subjected to a first format conversion to obtain a compressed video sequence; the data volume of the compressed video sequence is smaller than the data volume of the original video sequence.

[0012] In one embodiment, performing an optical flow alignment operation on a compressed video sequence according to a preset denoising model, and denoising the compressed video sequence after the optical flow alignment to obtain a denoised video sequence, includes:

[0013] Determine multiple groups of video sequences from the compressed video sequence; each group of video sequences includes any video frame in the compressed video sequence and at least one reference frame corresponding to the video frame;

[0014] According to the denoising model, optical flow alignment is performed on each group of video sequences, and each group of video sequences after optical flow alignment is denoised to obtain denoised video frames corresponding to the video frames in each group of video sequences;

[0015] According to the denoised video frames corresponding to the video frames in each group of video sequences, a denoised video sequence is obtained.

[0016] In one of the embodiments, the at least one reference frame includes a previous frame and a next frame of the video frame in the compressed video sequence.

[0017] In one embodiment, the denoising model includes an optical flow estimation model and a denoising model; optical flow alignment operations are performed on each group of video sequences according to the denoising model, and each group of video sequences after optical flow alignment is denoised to obtain denoised video frames corresponding to video frames in each group of video sequences, including:

[0018] For any set of video sequences, the optical flow field between each reference frame and the video frame in the video sequence is calculated according to the optical flow estimation model;

[0019] According to the optical flow field, each reference frame is aligned to the video frame to obtain aligned reference frames;

[0020] The video frames are denoised according to the denoising model and the aligned reference frames to obtain denoised video frames corresponding to the video frames in the video sequence.

[0021] In one embodiment, calculating the optical flow field between each reference frame and the video frame in the video sequence according to the optical flow estimation model includes:

[0022] Channel-compressing the video frames and reference frames in the video sequence to obtain single-channel video frames and reference frames;

[0023] Down-sampling the single-channel video frame and each reference frame to obtain a sampled video frame and each reference frame;

[0024] The sampled video frames and each reference frame are input into the optical flow estimation model to obtain the optical flow field between each reference frame in the video sequence and the video frame.

[0025] In one embodiment, denoising a video frame according to a denoising model and aligned reference frames to obtain a denoised video frame corresponding to a video frame in a video sequence includes:

[0026] The aligned reference frames and video frames are input into the denoising model to obtain denoised video frames corresponding to the video frames.

[0027] In one embodiment, obtaining a denoised video sequence according to denoised video frames corresponding to video frames in each group of video sequences includes:

[0028] Convert each denoised video frame into a second format to obtain a plurality of denoised video frames in the second format;

[0029] A denoised video sequence is determined based on a plurality of denoised video frames in a second format.

[0030] In one embodiment, the process of acquiring the optical flow estimation model includes:

[0031] Get the initial optical flow estimation model;

[0032] The initial optical flow estimation model is pruned and quantized to obtain an optical flow estimation model.

[0033] In one embodiment, the initial optical flow estimation model is pruned and quantized to obtain the optical flow estimation model, including:

[0034] Obtaining a preset first model sparsification degree;

[0035] According to the sparsification degree of the first model and the parameter values ​​of each parameter in the initial optical flow estimation model, the initial optical flow estimation model is pruned to obtain a pruned initial optical flow estimation model;

[0036] The pruned initial optical flow estimation model is quantized to obtain a quantized initial optical flow estimation model;

[0037] The quantized initial optical flow estimation model is determined as the optical flow estimation model.

[0038] In one embodiment, the training process of the denoising model includes:

[0039] Acquire noise reduction training data; the noise reduction training data includes a plurality of noise reduction training groups; each noise reduction training group includes a target noisy video frame and at least one reference noisy video frame aligned with the target noisy video frame;

[0040] Each denoising training group is input into the initial denoising model respectively, and the initial denoising model is iteratively trained through each denoising training group until the initial denoising model converges to obtain a denoising model.

[0041] In one embodiment, obtaining noise reduction training data includes:

[0042] Acquire a denoising training sample; the denoising training sample includes a plurality of consecutive noisy video frames and a corresponding plurality of consecutive noise-free video frames;

[0043] The denoising training samples are grouped to obtain a plurality of denoising sample groups; each denoising sample group includes a target noisy video frame and at least one reference noisy video frame;

[0044] Aligning each denoised sample group through an optical flow estimation model to obtain a plurality of aligned denoised sample groups;

[0045] The plurality of aligned denoised sample groups are determined as denoised training data.

[0046] In one embodiment, obtaining a denoising training sample includes:

[0047] Acquire an original data set; the original data set includes a plurality of consecutive noisy sample video frames and corresponding noise-free sample video frames;

[0048] Each sample video frame in the original data set is segmented to obtain a segmented original data set, and a denoising training sample is determined according to the segmented original data set.

[0049] In one embodiment, determining the denoising training sample according to the segmented original data set includes:

[0050] Perform data augmentation on the segmented original data set to obtain an augmented original data set;

[0051] The augmented original data set is determined as the denoising training sample.

[0052] In one embodiment, iteratively training the initial denoising model through each denoising training group until the initial denoising model converges includes:

[0053] The initial denoising model is trained through each denoising training group respectively, and the training group loss function value of the initial denoising model trained by each denoising training group is obtained;

[0054] The parameters of the initial denoising model are updated according to the loss function values ​​of each training group until the initial denoising model meets the preset conditions, and the initial denoising model is determined to converge.

[0055] In one embodiment, the initial denoising model is trained by each denoising training group respectively, and the training group loss function value of the initial denoising model trained by each denoising training group is obtained, including:

[0056] For any denoising training group, a predicted denoised video frame of a target noisy video frame in the denoising training group is obtained through an initial denoising model;

[0057] According to the target noisy video frame and the predicted denoised video frame in the denoising training group, a training group loss function value of the initial denoising model trained by the denoising training group is determined.

[0058] In one embodiment, after obtaining the noise reduction model, the method further includes:

[0059] Obtaining a preset second model sparsification degree;

[0060] According to the sparsification degree of the second model and the parameter value of each parameter in the denoising model, the denoising model is pruned to obtain a pruned denoising model;

[0061] The pruned denoising model is quantized to obtain a quantized denoising model; the quantized denoising model is used to predict a denoised video frame corresponding to the video frame.

[0062] In a second aspect, the present application also provides a video denoising device, comprising:

[0063] An acquisition module, used for acquiring an original video sequence;

[0064] A compression module, used for compressing the original video sequence to obtain a compressed video sequence;

[0065] The denoising module is used to perform optical flow alignment on the compressed video sequence according to a preset denoising model, and denoise the compressed video sequence after optical flow alignment to obtain a denoised video sequence; the denoising model is obtained by pruning and quantizing the initial denoising model.

[0066] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods provided in the embodiment of the first aspect are implemented.

[0067] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods provided in the embodiment of the first aspect above are implemented.

[0068] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the methods provided in the embodiments of the first aspect above.

[0069] The above-mentioned video denoising method, device, equipment, storage medium and program product obtain the original video sequence, and compress the original video sequence to obtain a compressed video sequence, and then perform an optical flow alignment operation on the compressed video sequence according to a preset denoising model, and denoise the compressed video sequence after the optical flow alignment to obtain a denoised video sequence; wherein the denoising model is obtained by pruning and quantizing the initial denoising model. In this method, since the original video sequence is compressed, the amount of data of the original video sequence is reduced, so that the amount of calculation when the compressed video sequence is denoised by the denoising model is less, thereby improving the denoising rate of the original video sequence and ensuring the high real-time performance of video processing; and, the denoising model is obtained by pruning and quantizing, so that the parameter scale and complexity in the denoising model are low, reducing the computational complexity of the denoising model, thereby improving the real-time performance of video processing; and, before denoising the compressed video sequence, optical flow estimation is performed first, further improving the accuracy of video denoising. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 A diagram showing an application environment of a video denoising method in an embodiment;

[0072] Figure 2 1 is a flow chart of a video denoising method according to an embodiment;

[0073] Figure 3 is a flow chart of a video denoising method according to another embodiment;

[0074] Figure 4 is a flow chart of a video denoising method according to another embodiment;

[0075] Figure 5 is a flow chart of a video denoising method according to another embodiment;

[0076] Figure 6 is a flow chart of a video denoising method according to another embodiment;

[0077] Figure 7 is a flow chart of a video denoising method according to another embodiment;

[0078] Figure 8 is a flow chart of a video denoising method according to another embodiment;

[0079] Fig. 9 is a flow chart of a video denoising method according to another embodiment;

[0080] Fig.10 is a flow chart of a video denoising method according to another embodiment;

[0081] Fig.11 is a flow chart of a video denoising method according to another embodiment;

[0082] Fig.12 is a flow chart of a video denoising method according to another embodiment;

[0083] Fig.13 is a flow chart of a video denoising method according to another embodiment;

[0084] Fig.14 is a flow chart of a video denoising method according to another embodiment;

[0085] Fig.15 is a flow chart of a video denoising method according to another embodiment;

[0086] Fig.16 is a flow chart of a video denoising method according to another embodiment;

[0087] Fig.17 is a flow chart of a video denoising method according to another embodiment;

[0088] Fig.18 is a structural block diagram of a video denoising device in one embodiment;

[0089] Fig.19 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0091] The video denoising method provided in the embodiment of the present application can be applied to a shooting device. The shooting device may include a camera and a computer device connected to the camera. The shooting device may be a shooting device with a camera function, a mobile phone or a tablet computer with a camera, etc. For example, the shooting device may be Figure 1 The motion camera 101. The computer device may be built in the shooting device, or may be connected to the outside of the shooting device; the computer device may be a terminal.

[0092] In an exemplary embodiment, Figure 2 As shown, a video denoising method is provided, which is applied to Figure 1 Taking the computer device in the sports camera as an example, the following steps are included:

[0093] S201, obtaining an original video sequence.

[0094] The original video sequence may be a video sequence shot in real time by a motion camera. The original video sequence shot in real time by the motion camera is denoised, and the denoised video sequence is displayed on the motion camera.

[0095] During the shooting process of the sports camera, the computer device can obtain the original video sequence in real time. The original video sequence can be the original video data of the digital signal converted by the camera of the sports camera into the captured light source signal. After acquiring the original video sequence, the camera can send the original video data to the computer device of the sports camera.

[0096] S202, compressing the original video sequence to obtain a compressed video sequence.

[0097] The format of the original video sequence shot by the motion camera is unprocessed and uncompressed, which contains more image information and richer tones. However, since the original video data includes more content, the data volume of the original video sequence is large.

[0098] Therefore, in order to improve the denoising speed of the original video sequence, the original video sequence may be firstly compressed; compressing the original video sequence may reduce the data volume of the original video sequence, and the compressed video sequence may be the compressed original video sequence.

[0099] Among them, the original video sequence can be compressed by a video encoder. Specifically, the video encoder can compress each frame of video in the original video sequence, thereby reducing the video file size of the original video sequence; the video encoder can compress the data of the original video sequence by adjusting the frame interval, encoding quality, video resolution, etc.

[0100] The original video sequence may also be compressed by a video compression model. Specifically, the original video sequence is input into the video compression model, and the original video sequence is compressed by the video compression model to obtain a compressed video sequence.

[0101] S203, performing an optical flow alignment operation on the compressed video sequence according to a preset denoising model, and denoising the compressed video sequence after the optical flow alignment to obtain a denoised video sequence; the denoising model is obtained by pruning and quantizing the initial denoising model.

[0102] The denoising model is a pre-trained model for denoising a video sequence. The process of obtaining the denoising model may include: training the initial neural network model through noise-free video sequences and noisy video sequences, and when the accuracy or number of iterations of the initial neural network model reaches a preset threshold, determining the initial neural network model as the initial denoising model.

[0103] After obtaining the initial denoising model, in order to improve the inference speed of the initial denoising model, the initial denoising model can be pruned and quantized, and the pruned and quantized initial denoising model is determined as the denoising model; wherein the model inference speed of the denoising model is greater than the model inference speed of the initial denoising model, and the accuracy of the denoising model is greater than or equal to the accuracy of the initial denoising model, and the accuracy threshold can be 99%.

[0104] The compressed video sequence obtained in the above embodiment is input into the denoising model, the compressed video sequence is subjected to optical flow alignment operation through the denoising model, and the compressed video sequence after optical flow alignment is denoised to obtain the denoised video sequence output by the denoising model.

[0105] The video denoising method provided in the embodiment of the present application obtains the original video sequence, and compresses the original video sequence to obtain a compressed video sequence, and then performs an optical flow alignment operation on the compressed video sequence according to a preset denoising model, and denoises the compressed video sequence after the optical flow alignment to obtain a denoised video sequence; wherein the denoising model is obtained by pruning and quantizing the initial denoising model. In this method, since the original video sequence is compressed, the amount of data of the original video sequence is reduced, so that the amount of calculation when the compressed video sequence is denoised by the denoising model is less, thereby improving the denoising rate of the original video sequence and ensuring the high real-time performance of video processing; and the denoising model is obtained by pruning and quantizing, so that the parameter scale and complexity in the denoising model are low, reducing the computational complexity of the denoising model, thereby improving the real-time performance of video processing; and before denoising the compressed video sequence, optical flow estimation is performed first, further improving the accuracy of video denoising.

[0106] The following describes how to compress an original video sequence through an embodiment. In an exemplary embodiment, the original video sequence is compressed to obtain a compressed video sequence, including: performing a first format conversion on the original video sequence to obtain a compressed video sequence; the data volume of the compressed video sequence is smaller than the data volume of the original video sequence.

[0107] The original video sequence may be converted into the first format by performing a first format conversion, and then the original video sequence after the first format conversion is determined as a compressed video sequence, wherein the data volume of the compressed video sequence in the first format is smaller than the data volume of the original video sequence.

[0108] Taking the 16-bit RAW format of the original video sequence arranged in RGrGbB as an example, the single-channel Raw image can be converted into four channels (R, Gr, Gb, B), and then the Gr and Gb channels are averaged; it can also be obtained in YUV format or RGB format through image signal processing (ISP).

[0109] Among them, the video in RAW format is the original video data without compression, and each pixel in the video includes complete chrominance and brightness information, which leads to a large amount of video data in RAW format. In the YUV format, "Y" represents brightness (Luminance), that is, grayscale value; "U" and "V" represent chrominance (Chrominance), which is used to describe the color and saturation of the image and is used to specify the color of the pixel. RGB is the most common color standard in the industry. Various colors are obtained by changing the three color channels of red, green and blue and superimposing them with each other. RGB almost includes all colors that can be perceived by human vision. In this way, the compressed video frame format can significantly reduce the amount of data stored and transmitted. Among them, taking the format of the original video sequence in RAW format and the first format in RGB format as an example, the way to convert the original video sequence into the first format can be to determine the pixel arrangement and color encoding method of the original video sequence, and parse the original video sequence into RGB color components according to the pixel arrangement and color encoding method of the original video sequence.

[0110] The video denoising method provided in the embodiment of the present application performs a first format conversion on the original video sequence to obtain a compressed video sequence; wherein the data volume of the compressed video sequence is less than the data volume of the original video sequence. In this method, the purpose of compressing the video is achieved by performing format conversion on the original video sequence, thereby improving the efficiency of data compression on the original video sequence; and, all pixels in the original video sequence are retained, which will not cause distortion of the compressed video sequence, thereby ensuring the accuracy of the compressed video sequence.

[0111] In one embodiment, Figure 3 As shown, an optical flow alignment operation is performed on a compressed video sequence according to a preset denoising model, and the compressed video sequence after the optical flow alignment is denoised to obtain a denoised video sequence, including the following steps:

[0112] S301, determining multiple groups of video sequences from a compressed video sequence; each group of video sequences includes any video frame in the compressed video sequence and at least one reference frame corresponding to the video frame.

[0113] For any video frame in the compressed video sequence, at least one reference frame corresponding to the video frame can be obtained from the compressed video sequence, and the video frame and the at least one reference frame corresponding to the video frame are determined as a group of video sequences.

[0114] The at least one reference frame corresponding to the video frame may be an adjacent frame of the video frame or a frame at a preset interval with the video frame. Optionally, the at least one reference frame includes a previous frame and a next frame of the video frame in the compressed video sequence.

[0115] Therefore, each frame in the compressed video sequence can be determined as a corresponding video sequence. For example, the compressed video sequence may include 4 frames: the first frame, the second frame, the third frame and the fourth frame, and the first frame, the first frame and the second frame can be used as the video sequence corresponding to the first frame, and the first frame and the second frame are reference frames of the first frame; the first frame, the second frame and the third frame are determined as the video sequence corresponding to the second frame, and the first frame and the third frame are reference frames of the second frame; the second frame, the third frame and the fourth frame are determined as the video sequence corresponding to the third frame, and the second frame and the fourth frame are reference frames of the third frame; the third frame, the fourth frame and the fourth frame are determined as the video sequence corresponding to the fourth frame, and the third frame and the fourth frame are reference frames of the fourth frame.

[0116] S302, performing optical flow alignment operations on each group of video sequences according to the denoising model, and denoising each group of video sequences after the optical flow alignment, to obtain denoised video frames corresponding to the video frames in each group of video sequences.

[0117] Based on the multiple groups of video sequences obtained above, each group of video sequences can be input into the denoising model respectively, and the optical flow alignment operation is performed on each group of video sequences through the denoising model. Then, each group of video sequences after the optical flow alignment is denoised to determine the denoised video frames corresponding to the video frames in each group of video sequences.

[0118] For example, if the video sequence is the first frame, the second frame and the third frame, then the optical flow alignment operation is performed on the video sequence according to the denoising model to obtain the optical flow field of the first frame relative to the second frame, and the optical flow field of the third frame relative to the second frame, and the first frame and the third frame are aligned to the second frame according to the two optical flow fields to obtain the aligned video sequence, and the aligned video sequence is denoised to obtain the denoised video frame corresponding to the second frame. The denoised video frame is the noise-free video frame corresponding to the second frame.

[0119] S303: Obtain a denoised video sequence according to the denoised video frames corresponding to the video frames in each group of video sequences.

[0120] The denoised video frames corresponding to the video frames in each group of video sequences are spliced ​​according to the acquisition time to obtain a denoised video sequence, which includes the denoised video corresponding to each frame of the original video sequence.

[0121] The video denoising method provided in the embodiment of the present application determines multiple groups of video sequences from a compressed video sequence; each group of video sequences includes any video frame in the compressed video sequence, and at least one reference frame corresponding to the video frame, and performs optical flow alignment operations on each group of video sequences according to the denoising model, and denoises each group of video sequences after the optical flow alignment to obtain denoised video frames corresponding to the video frames in each group of video sequences, and then obtains the denoised video sequence based on the denoised video frames corresponding to the video frames in each group of video sequences. In this method, when denoising the compressed video sequence, each frame of video is denoised separately, so that each frame of video is denoised separately in a targeted manner, thereby improving the accuracy of denoising; and when denoising the video frame, the reference frame corresponding to the video frame is also considered, thereby further improving the accuracy of video frame denoising.

[0122] In one embodiment, Figure 4 As shown, the denoising model includes an optical flow estimation model and a denoising model; optical flow alignment operations are performed on each group of video sequences according to the denoising model, and each group of video sequences after optical flow alignment is denoised to obtain denoised video frames corresponding to video frames in each group of video sequences, including the following steps:

[0123] S401 , for any group of video sequences, calculating the optical flow field between each reference frame and the video frame in the video sequence according to the optical flow estimation model.

[0124] The optical flow field may be a motion pattern of pixels caused by the movement of an object or a camera in consecutive video frames, and the optical flow field may be used to describe the motion vector of each pixel between adjacent frames.

[0125] For any set of video sequences, the optical flow field can be calculated by a pre-trained optical flow estimation model; specifically, each set of video sequences can be input into the optical flow estimation model, and each set of video sequences can be analyzed by the optical flow estimation model to obtain the optical flow field between each reference frame and the video frame in each set of video sequences. The optical flow field between each reference frame and the video frame can include the position offset of each reference frame relative to the same pixel point in the video frame.

[0126] For example, a video sequence includes a first frame, a second frame and a third frame, the second frame is a video frame, and the first frame and the third frame are reference frames of the second frame. Therefore, the first frame, the second frame and the third frame are input into the optical flow estimation model to obtain the optical flow fields between the first frame and the third frame and the second frame respectively.

[0127] In one embodiment, when the second frame is used as the video frame, the optical flow fields between the first frame and the third frame and the second frame are calculated respectively; then, when the third frame is used as the video frame, the optical flow fields between the second frame and the fourth frame and the third frame are calculated respectively. At this time, since the optical flow field between the third frame and the second frame has been calculated, when the third frame is used as the video frame, it is only necessary to negate the optical flow field between the third frame and the second frame to obtain the reverse optical flow field, that is, the optical flow field from the second frame to the third frame. There is no need to repeat the calculation. At this time, the number of optical flow calculations is almost halved.

[0128] S402: align each reference frame to the video frame according to the optical flow field to obtain aligned reference frames.

[0129] According to the optical flow field between each reference frame and the video frame in the video sequence, the pixels in each reference frame are aligned to the positions of corresponding pixels in the video frame to obtain aligned reference frames.

[0130] For example, continuing to use the video sequence including the first frame, the second frame and the third frame as an example, based on the optical flow fields between the first frame and the third frame and the second frame respectively, the first frame is aligned to the second frame, and the third frame is aligned to the second frame to obtain the aligned first frame and third frame.

[0131] S403, denoising the video frames according to the denoising model and the aligned reference frames to obtain denoised video frames corresponding to the video frames in the video sequence.

[0132] Since the reference frame is an adjacent video frame of the corresponding video frame, the video frame can be denoised by using the reference frame aligned with the video frame; specifically, each reference frame and video frame can be input into a denoising model, and each reference frame and video frame can be analyzed by the denoising model to obtain a denoised video frame corresponding to the video frame in the video sequence.

[0133] Each reference frame and video frame can be input into the denoising model to obtain a denoised video frame corresponding to the video frame in the video sequence; wherein the execution steps of the denoising model may include: obtaining the reference denoised frames of each aligned reference frame and the denoised frames of the video frame, and then determining the pixel average value corresponding to each pixel point in each reference denoised frame and the denoised frame as the denoised video frame corresponding to the video frame.

[0134] The video denoising method provided in the embodiment of the present application calculates the optical flow field between each reference frame in the video sequence and the video frame according to the optical flow estimation model for any group of video sequences, and aligns each reference frame to the video frame according to the optical flow field to obtain aligned reference frames, and then denoises the video frame according to the denoising model and the aligned reference frames to obtain denoised video frames corresponding to the video frames in the video sequence. In the method, since the denoising model is pre-trained and a denoising model is used for each group of video sequences, the denoising effect of each video frame can be guaranteed by denoising each group of video sequences through the optical flow estimation model and the denoising model in the denoising model; and the optical flow estimation model and the denoising model are pruned and quantized models, and the computational complexity of the optical flow estimation model and the denoising model is low, so that the speed of denoising the video frames is faster, thereby improving the real-time performance of video processing; and before denoising the video frames, the video frames and the reference frames are aligned so that the video frames and the reference frames are in the same dimension, thus ensuring the reliability of denoising the video frames through the reference frames and improving the accuracy of denoising.

[0135] In an exemplary embodiment, Figure 5 As shown, according to the optical flow estimation model, the optical flow field between each reference frame and the video frame in the video sequence is calculated, including the following steps:

[0136] S501 , performing channel compression on video frames and reference frames in a video sequence to obtain single-channel video frames and reference frames.

[0137] Before calculating the optical flow field between each reference frame and the video frame, the video frames and each reference frame in the video sequence may be channel compressed to obtain single-channel video frames and each reference frame.

[0138] Since the video frames and reference frames in the video sequence can be in RGB format, the video frames and reference frames can be three-channel data; the channel compression method for the video frames and reference frames can be to take the average of the three-channel data of the video frames and reference frames, respectively, to obtain a single-channel video frame and a single-channel grayscale image of each reference frame. For example, for the video frame, the R component, G component and B component of each pixel in the video frame are averaged to obtain a single-channel value of each pixel, thereby obtaining a single-channel video frame.

[0139] S502 , down-sample the single-channel video frame and each reference frame to obtain a sampled video frame and each reference frame.

[0140] After obtaining the single-channel video frames and each reference frame, in order to further improve the speed of calculating the optical flow field, the single-channel video frames and each reference frame can also be downsampled to obtain low-resolution video frames and each reference frame, that is, sampled video frames and each reference frame.

[0141] Among them, the single-channel video frames and each reference frame can be downsampled by average pooling, maximum pooling, bilinear interpolation, etc. It should be noted that the downsampling method can be determined according to application needs and performance requirements.

[0142] S503: Input the sampled video frames and each reference frame into an optical flow estimation model to obtain optical flow fields between each reference frame in the video sequence and the video frame.

[0143] The sampled video frames and each reference frame are input into the optical flow estimation model, and the sampled video frames and each reference frame are analyzed by the optical flow estimation model. The optical flow estimation model outputs the optical flow field between each reference frame in the video sequence and the video frame.

[0144] In the optical flow estimation model, for the calculation of the optical flow field between any reference frame and the video frame, the optical flow field between the sampled reference frame and the sampled video frame can be calculated based on the brightness optical flow method, and the optical flow field between the sampled reference frame and the sampled video frame can be determined as the optical flow field between the corresponding reference frame and the video frame.

[0145] Among them, the brightness-based optical flow method includes finding corresponding pixel points in the sampled reference frame and the sampled video frame, calculating the motion vectors between the corresponding pixel points respectively, and determining the optical flow field between the corresponding reference frame and the video frame based on the motion vectors between the corresponding pixel points.

[0146] Optionally, a gradient-based optical flow method may also be used to calculate the optical flow fields between each reference frame in the video sequence and the video frame.

[0147] The video denoising method provided by the embodiment of the present application performs channel compression on the video frames and each reference frame in the video sequence to obtain the single-channel video frames and each reference frame, and downsamples the single-channel video frames and each reference frame to obtain the sampled video frames and each reference frame, and then inputs the sampled video frames and each reference frame into the optical flow estimation model to obtain the optical flow field between each reference frame in the video sequence and the video frame. In this method, before calculating the optical flow field between each reference frame and the video frame, the video frame and each reference frame are first channel compressed and downsampled, and then the optical flow field is calculated through the optical flow estimation model, which reduces the amount of data when calculating the optical flow field between each reference frame and the video frame, thereby improving the speed of calculating the optical flow field and reducing the time consumed in calculating the optical flow field.

[0148] In an exemplary embodiment, video frames are denoised according to a denoising model and aligned reference frames to obtain denoised video frames corresponding to video frames in a video sequence, including: inputting the aligned reference frames and the video frames into the denoising model to obtain denoised video frames corresponding to the video frames.

[0149] The aligned parameter frames and video frames are connected together to finally obtain data of multiple channels, and the data of multiple channels are input into the denoising model. The data of multiple channels are analyzed by the denoising model to obtain denoised video frames corresponding to the video frames.

[0150] Among them, each aligned reference frame and video frame can be RGB three-channel data. Therefore, each aligned reference frame and video frame can be channel-connected to obtain multi-channel data, and the multi-channel data is input into the denoising model. Therefore, inputting each aligned reference frame and video frame into the denoising model is essentially inputting 9-channel data into the denoising model.

[0151] The denoising model input is 3*(1+number of reference frames) channels. For example, 1 reference frame is 6 channels, and 2 reference frames are 9 channels.

[0152] Optionally, the denoising model can be obtained by training a current mainstream denoising network or a lightweight denoising network suitable for inference on a mobile terminal.

[0153] The video denoising method provided in the embodiment of the present application inputs the aligned reference frames and the video frame into the denoising model to obtain the denoised video frame corresponding to the video frame. In this method, the video frame is assisted in denoising by the reference frames of the video frame, which helps the denoising model to more accurately separate the noise and the signal, thereby improving the denoising effect.

[0154] The denoised video frame output by the above denoising model may be three-channel data in RGB format. Therefore, the three-channel data in RGB format needs to be converted into a video format that can be displayed by a shooting device. This is described in detail below through an embodiment. In an exemplary embodiment, Figure 6 As shown, according to the denoised video frames corresponding to the video frames in each group of video sequences, a denoised video sequence is obtained, which includes the following steps:

[0155] S601: Convert each denoised video frame into a second format to obtain a plurality of denoised video frames in the second format.

[0156] The second format may be a video format supported by the shooting device, for example, the second format may be an MP4 format.

[0157] The denoised video frames compressed by each channel can be converted to a second format through the format conversion model to obtain multiple denoised video frames in the second format. For example, for any denoised video frame after compression, the denoised video frame compressed by the channel is input into the format conversion model, and the denoised video frame compressed by the channel is format converted through the format conversion model to obtain the corresponding denoised video frame in the second format.

[0158] Taking the denoised video frame after channel compression as an RGB format and the second format as an MP4 format as an example, the format conversion model is a model that supports converting the RGB format into the MP4 format.

[0159] Optionally, the denoised video frame in RGB format may also be written into a pre-created MP4 file to obtain the denoised video frame in MP4 format.

[0160] S603: Determine a denoised video sequence according to the plurality of denoised video frames in the second format.

[0161] The plurality of denoised video frames in the second format are sequentially spliced ​​according to the acquisition time to obtain a denoised video sequence; the denoised video sequence includes all the denoised video frames in the second format.

[0162] In the above embodiment, each denoised video frame in RGB format may be first converted into a YUV format video frame, and then color adjustment, sharpening enhancement and other operations may be performed on the YUV video frame, and then the YUV video frame after color adjustment and sharpening enhancement may be converted into MP4 format, and then the video frames in MP4 format may be spliced ​​in sequence according to the acquisition time to obtain a denoised video sequence.

[0163] The video denoising method provided in the embodiment of the present application converts each denoised video frame into a second format to obtain multiple denoised video frames in the second format, and then determines a denoised video sequence based on the multiple denoised video frames in the second format. In this method, in order to better display it in a shooting device, the denoised video frame is converted into a video format supported by the shooting device, so that the denoised video sequence is more realistic and has better video quality.

[0164] In one embodiment, Figure 7 As shown, the acquisition process of the optical flow estimation model includes the following steps:

[0165] S701, obtaining an initial optical flow estimation model.

[0166] The initial optical flow estimation model may be trained based on a basic optical flow estimation network. The training process for constructing the initial optical flow estimation model will not be described in detail in the embodiment of the present application, and the model may be obtained through basic training methods.

[0167] S702, pruning and quantizing the initial optical flow estimation model to obtain an optical flow estimation model.

[0168] In order to enable the optical flow estimation model to be deployed in resource-limited embedded devices, such as motion cameras, the parameter scale of the optical flow estimation model can be reduced and the computational complexity can be reduced while maintaining the performance of the optical flow estimation model. Therefore, the initial optical flow estimation model can be pruned and quantized.

[0169] In an exemplary embodiment, Figure 8 As shown, the initial optical flow estimation model is pruned and quantized to obtain an optical flow estimation model, including the following steps:

[0170] S801, obtaining a preset first model sparsification degree.

[0171] Among them, the degree of model sparsity can represent the degree of sparsity of the parameters in the model. Model sparsity can reduce the complexity and storage requirements of the model, improve the generalization ability and computing efficiency of the model, thereby reducing the amount of calculation and improving the reasoning speed.

[0172] The first model sparsification degree may represent the sparsification degree of the optical flow estimation model; the first model sparsification degree may include a sparsification ratio, for example, 80%, which represents the ratio of non-zero parameters in the model; a higher first model sparsification degree indicates that more parameters in the optical flow estimation model are 0.

[0173] S802: Perform a pruning operation on the initial optical flow estimation model according to the sparsification degree of the first model and the parameter value of each parameter in the initial optical flow estimation model to obtain a pruned initial optical flow estimation model.

[0174] Get the parameter value of each parameter in the initial optical flow estimation model. The larger the parameter value, the more important the corresponding parameter is in the initial optical flow estimation model.

[0175] Therefore, according to the sparse ratio corresponding to the sparsification degree of the first model, the smaller parameter value can be determined as 0, thereby pruning the initial optical flow estimation model to obtain the pruned initial optical flow estimation model.

[0176] Specifically, according to the sparsification degree of the first model and the total number of parameters in the initial optical flow estimation model, the number of parameters that need to be pruned in the initial optical flow estimation model is determined, and the parameter values ​​of the parameters with smaller number of parameters are determined to be 0. For example, if the initial optical flow estimation model includes 10 parameters and the sparsification degree of the first model is 70%, the parameter values ​​of the smallest 3 parameters among the 10 parameters are determined to be 0.

[0177] S803, quantizing the pruned initial optical flow estimation model to obtain a quantized initial optical flow estimation model.

[0178] The pruned initial optical flow estimation model is fine-tuned through optical flow training samples, and it is determined whether the model size, inference speed and accuracy of the fine-tuned initial optical flow estimation model meet the requirements. If not, the initial optical flow estimation model is pruned. If it meets the requirements, the fine-tuned initial optical flow estimation model is determined as the quantized optical flow estimation model.

[0179] Among them, model quantization is essentially to represent the high-bit weights and eigenvalues ​​in the model with lower bits; that is, the data format (32-bit double-precision floating point number) in the fine-tuned initial optical flow estimation model can be adjusted to 8-bit or 16-bit fixed-point number, and then the initial optical flow estimation model with the adjusted format is retrained through optical flow training samples (training-perceptual quantization) to obtain the quantized initial optical flow estimation model.

[0180] The quantized optical flow estimation model is a low-bit quantized inference model.

[0181] S804: Determine the quantized initial optical flow estimation model as the optical flow estimation model.

[0182] The quantized initial optical flow estimation model is determined as the optical flow estimation model, and the optical flow estimation model is used to predict the optical flow field between video frames.

[0183] The video denoising method provided in the embodiment of the present application obtains a preset first model sparsification degree, and performs a pruning operation on the initial optical flow estimation model according to the first model sparsification degree and the parameter values ​​of each parameter in the initial optical flow estimation model to obtain a pruned initial optical flow estimation model, and then performs model quantization on the pruned initial optical flow estimation model to obtain a quantized initial optical flow estimation model, and finally determines the quantized initial optical flow estimation model as the optical flow estimation model. In this method, since quantizing and pruning the initial optical flow estimation model can reduce the number of parameters in the model and reduce the computational complexity of the model, the optical flow estimation model obtained after pruning and quantization is used to predict the optical flow field between video frames, thereby improving the speed of predicting the optical flow field and thus improving the speed of video denoising.

[0184] The following is an example of an embodiment to illustrate the training process of the denoising model. In an exemplary embodiment, Fig. 9 As shown in Figure 1, the training process of the denoising model includes the following steps:

[0185] S901, acquiring noise reduction training data; the noise reduction training data includes a plurality of noise reduction training groups; each noise reduction training group includes a target noisy video frame and at least one reference noisy video frame aligned with the target noisy video frame.

[0186] In an exemplary embodiment, Fig.10 As shown, obtaining denoising training data includes the following steps:

[0187] S1001, obtaining a noise reduction training sample; the noise reduction training sample includes a plurality of consecutive noisy video frames and a corresponding plurality of consecutive noise-free video frames.

[0188] The noisy video frames and the corresponding noise-free video frames collected in the historical time period may be directly obtained, and the noisy video frames and the corresponding noise-free video frames collected in the historical time period may be determined as noise reduction training samples.

[0189] S1002, grouping the denoising training samples to obtain a plurality of denoising sample groups; each denoising sample group includes a target noisy video frame and at least one reference noisy video frame.

[0190] The denoising training samples may be grouped to obtain a plurality of denoising sample groups by dividing the denoising sample groups into a preset number of noisy video frames from a plurality of consecutive noisy video frames in the denoising training samples, wherein each denoising sample group includes a target noisy video frame and at least one reference noisy video frame corresponding to the target noisy video frame.

[0191] The target noisy video frame may be any noisy video frame in the noise reduction sample group, and the at least one reference noisy video frame may include a previous frame and a next frame of the target noisy video frame.

[0192] Taking the example that the denoising training samples include 10 frames of noisy video frames, the denoising sample group may include: the 1st, 2nd and 3rd noisy video frames are determined as a group of denoising sample groups, the 2nd, 3rd and 4th frames are determined as a group of denoising sample groups, ..., and so on, to obtain multiple denoising sample groups.

[0193] S1003, aligning each denoised sample group using an optical flow estimation model to obtain a plurality of aligned denoised sample groups.

[0194] For any denoised sample group, the denoised sample group is input into an optical flow estimation model to obtain an optical flow field between each reference noisy video frame and a target noisy video frame in the denoised sample group; according to the optical flow field between each reference noisy video frame and the target noisy video frame, each reference noisy video frame is aligned to the target noisy video frame to obtain aligned reference noisy video frames; and each aligned reference noisy video frame and the target noisy video frame are determined as an aligned denoised sample group.

[0195] S1004: Determine the plurality of aligned denoised sample groups as denoised training data.

[0196] Determine multiple aligned noise-reduced sample groups as noise reduction training data; the noise reduction training data can be constructed from multiple consecutive video frames.

[0197] S902. Input each noise reduction training group into the initial noise reduction model respectively, and perform iterative training on the initial noise reduction model through each noise reduction training group until the initial noise reduction model converges to obtain a noise reduction model.

[0198] The condition for the initial noise reduction model to converge can be that the number of iterations of the initial noise reduction model reaches a preset number of iterations, or the accuracy rate of the initial noise reduction model reaches a preset accuracy rate threshold.

[0199] Specifically, if the convergence condition is that the number of iterations of the initial optical flow model reaches a preset number of iterations, input each noise reduction training group into the initial noise reduction model respectively, and train the initial noise reduction model according to each noise reduction training group until the number of iterations of the initial noise reduction model reaches the preset number of iterations, and determine the initial noise reduction model corresponding to the preset number of iterations as the noise reduction model.

[0200] If the convergence condition is that the accuracy rate of the initial noise reduction model reaches a preset accuracy rate threshold, input each noise reduction training group into the initial noise reduction model, and train the initial noise reduction model according to each noise reduction training group. During the training process of the initial noise reduction model, calculate the prediction accuracy rate of the initial noise reduction model. If the accuracy rate of the initial noise reduction model reaches the preset accuracy rate threshold, then determine the initial noise reduction model corresponding to when the accuracy rate threshold is reached as the noise reduction model; if the accuracy rate of the initial noise reduction model does not reach the preset accuracy rate threshold, then continue to train the initial noise reduction model through each noise reduction training group until the initial noise reduction model reaches the preset accuracy rate threshold.

[0201] The video denoising method provided by the embodiments of the present application obtains noise reduction training data; the noise reduction training data includes multiple noise reduction training groups; each noise reduction training group includes a target noisy video frame and at least one reference noisy video frame aligned with the target noisy video frame; input each noise reduction training group into the initial noise reduction model respectively, and perform iterative training on the initial noise reduction model through each noise reduction training group until the initial noise reduction model converges to obtain a noise reduction model. In this method, training the initial noise reduction model with multiple noise reduction training groups improves the accuracy and reliability of the noise reduction model training; moreover, setting the iterative convergence condition of the initial noise reduction model enables the initial noise reduction model to stop training when the iterative convergence condition is reached, thereby saving training time and computing resources and improving the training speed of the noise reduction model.

[0202] In an exemplary embodiment, as Fig.11 shown, obtaining noise reduction training samples includes the following steps:

[0203] S1101. Obtain the original data set, where the original data set includes multiple consecutive noisy sample video frames and corresponding noise-free sample video frames.

[0204] Among them, obtain multiple consecutive noisy video frames collected by the shooting device at historical moments, and then crop each noisy video frame to obtain multiple noisy video frames of the target size; denoise each noisy video frame of the target size according to the denoising algorithm to obtain multiple noise-free video frames of the target size.

[0205] Optionally, each noise-free video frame can also be randomly blurred and degraded by Gaussian blurs with different kernel sizes to obtain a blurred video sequence, and then obtain the noise parameters of the shooting device, and based on the noise parameters, add random noise to the blurred video sequence at different sensitivities (International Organization for Standardization, ISO) to obtain noisy video frames of the target size.

[0206] Then, the video sequence collected by the shooting device can be a Raw video sequence, and the noisy video frames of the target size and the noise-free video frames of the target size can be processed through an Image Singal Process (ISP) algorithm to obtain an RGB-format video sequence, that is, the original data set (multiple consecutive noisy sample video frames and corresponding noise-free sample video frames).

[0207] Among them, the cropping method can be central cropping or random cropping, etc.; the target size of each video frame in the noise reduction training samples can be 4K (3840*2060).

[0208] S1102. Segment each sample video frame in the original data set respectively to obtain the segmented original data set, and determine the noise reduction training samples according to the segmented original data set.

[0209] To improve the model training speed, each sample video frame in the original data set can be segmented to obtain the segmented original data set; then the segmented original data set can be determined as the noise reduction training samples.

[0210] The method of segmenting each sample video frame can be that for any sample video frame, the sample video frame is segmented into a preset number of small blocks; among them, the sample video frame can be evenly segmented, that is, the size of each small block of each sample video frame is the same.

[0211] The video denoising method provided in the embodiment of the present application obtains an original data set; wherein the original data set includes a plurality of continuous noisy sample video frames and corresponding noise-free sample video frames, and each sample video frame in the original data set is segmented to obtain a segmented original data set, and a denoising training sample is determined based on the segmented original data set. In this method, each sample video frame is segmented to enrich the denoising training samples of the initial denoising model, and the initial denoising model is trained using more abundant denoising training samples, so that the denoising model obtained by training is more accurate.

[0212] In order to improve the diversity of denoising training samples, data augmentation can be performed on the segmented original data set, which is described in detail below through an embodiment. In an exemplary embodiment, Fig.12 As shown, determining the denoising training sample according to the segmented original data set includes the following steps:

[0213] S1201, performing data augmentation on the segmented original data set to obtain an augmented original data set.

[0214] The method of data augmentation for the segmented original data set may include at least the following methods: randomly performing a time-sequence flip on each sample video frame in the segmented original data set, for example, flipping frames 1, 2, and 3 to frames 3, 2, and 1; randomly flipping each sample video frame in the segmented original data set in the horizontal and vertical directions; if the length and width of each sample video frame in the segmented original data set are the same, each sample video frame in the segmented original data set may be randomly rotated.

[0215] Data augmentation may be performed on the segmented original data set in at least one of the above-mentioned ways to obtain an augmented original data set.

[0216] S1202: Determine the augmented original data set as a denoising training sample.

[0217] The above-mentioned augmented original data set is determined as the denoising training sample, that is, the denoising training sample includes the original data set after brightening.

[0218] The video denoising method provided in the embodiment of the present application performs data augmentation on the segmented original data set to obtain an augmented original data set, and determines the augmented original data set as a denoising training sample. In this method, by performing data augmentation on the segmented original data set, more original data sets are generated, the training samples of the initial denoising model are expanded, the model is helped to learn and generalize better, the risk of overfitting of the initial denoising model is eliminated, and the accuracy of the denoising model is improved.

[0219] In an exemplary embodiment, Fig.13As shown, the initial denoising model is iteratively trained through each denoising training group until the initial denoising model converges, including the following steps:

[0220] S1301: Train the initial denoising model through each denoising training group respectively, and obtain the training group loss function value of the initial denoising model trained by each denoising training group.

[0221] The training group loss function value is an indicator used to measure the difference between the predicted value and the true value. During the training process, the parameters of the initial denoising model can be optimized by minimizing the training group loss function value, so that the initial denoising model can achieve more accurate predictions.

[0222] For any denoising training group, the denoising training group is input into the initial denoising model, the denoising training group is analyzed by the initial denoising model, the predicted value of the denoising training group is determined, the predicted value and the true value of the denoising training group are substituted into the preset loss function, and the training group loss function value of the initial denoising model trained by the denoising training group is obtained.

[0223] S1302, updating the parameters of the initial denoising model according to the loss function values ​​of each training group until the initial denoising model meets a preset condition, and determining that the initial denoising model converges.

[0224] In one embodiment, the gradient is calculated based on the training group loss function value of the initial denoising model for each training, the gradient is transferred back to each parameter of the initial denoising model using a back propagation algorithm, and then the parameters of the initial denoising model are updated according to an optimization algorithm (gradient descent method).

[0225] During the iterative training process, the loss function value in the initial denoising model can gradually decrease as the initial denoising model is trained. When the initial denoising model converges, the loss function value of the training group will tend to be stable, indicating that the initial denoising model has achieved a good prediction effect.

[0226] Therefore, the initial denoising model meets the preset condition that the training group loss function value of the initial denoising model is within a preset range; for example, the training group loss function value is less than a preset loss threshold.

[0227] In one embodiment, during the training process of the initial denoising model, each time the denoising training group is trained once, a training group loss function value obtained by each training of the initial denoising model can be obtained.

[0228] In one embodiment, the initial denoising model meets the preset conditions including that the number of iterative trainings of the initial denoising model reaches a preset number, and the accuracy of the output result of the initial denoising model reaches a preset accuracy threshold.

[0229] When the training of the initial denoising model reaches a preset number of iterations and the accuracy of the output result of the initial denoising model also reaches the accuracy threshold, the parameters in the initial denoising model corresponding to the maximum output result accuracy in the preset number of iterations are determined as the parameters of the denoising model.

[0230] Optionally, during the training process of the initial denoising model, each time the denoising training group is trained once, the initial denoising model can be tested with a test set, and the accuracy of the initial denoising model can be determined based on the prediction results of the initial denoising model on the test set and the actual results of the test set.

[0231] Among them, if the accuracy of the output result of the initial denoising model does not reach the preset accuracy threshold after the initial denoising model iterates a preset number of times, the initial denoising model can be iterated for a preset number of times until the accuracy of the output result of the initial denoising model reaches the preset accuracy threshold.

[0232] The video denoising method provided in the embodiment of the present application trains the initial denoising model through each denoising training group, obtains the training group loss function value of the initial denoising model trained by each denoising training group, and updates the parameters of the initial denoising model according to the loss function value of each training group, until the initial denoising model reaches the preset conditions, and determines that the initial denoising model converges. In this method, the initial denoising model is trained by multiple denoising training groups respectively, so that the initial denoising model is trained in batches, which reduces the video memory space used by the computer equipment during model training and improves the data processing speed; and avoids the volatility of the initial denoising model trained by individual denoising training groups, thereby improving the reliability of the denoising model.

[0233] In an exemplary embodiment, Fig.14 As shown, the initial denoising model is trained by each denoising training group respectively, and the training group loss function value of the initial denoising model trained by each denoising training group is obtained, including the following steps:

[0234] S1401: For any denoising training group, obtain a predicted denoised video frame of a target noisy video frame in the denoising training group through an initial denoising model.

[0235] The target noisy video frame in the denoising training group is denoised by the initial denoising model to obtain a predicted denoised video frame corresponding to the target noisy video frame.

[0236] It should be noted that the embodiment of the present application does not limit the specific denoising method in the initial denoising model.

[0237] S1402: Determine a training group loss function value of an initial denoising model trained by the denoising training group according to a target noisy video frame and a predicted denoised video frame in the denoising training group.

[0238] Among them, according to the preset loss function, the pixel values ​​of the target noisy video frame and the predicted denoised video frame in the denoising training group can be substituted into the corresponding loss function to obtain the training group loss function value of the initial denoising model trained by the denoising training group.

[0239] In selecting the loss function of the initial denoising model, a weighted average of multiple loss values ​​can be used to balance noise removal, detail retention, and consistency in the time domain.

[0240] The video denoising method provided in the embodiment of the present application, for any denoising training group, obtains the predicted denoised video frame of the target noisy video frame in the denoising training group through the initial denoising model, and determines the training group loss function value of the initial denoising model trained by the denoising training group according to the target noisy video frame and the predicted denoised video frame in the denoising training group. In this method, the training group loss function is calculated by the predicted denoised video frame of the target noisy video frame, which can ensure that the training group loss function value is more accurate, thereby improving the accuracy of the denoising model.

[0241] In order to enable the denoising model to be deployed in an embedded device with limited resources, the parameter scale of the denoising model can be reduced and the computational complexity can be reduced while maintaining the performance of the denoising model. Therefore, the model can be compressed and quantized. This is described in detail below through an embodiment. In one embodiment, Fig.15 As shown, after the noise reduction model is obtained, the embodiment includes the following:

[0242] S1501, obtaining a preset second model sparsification degree.

[0243] The second model sparsification degree may represent the sparsification degree of the denoising model; the second model sparsification degree may include a sparsification ratio, for example, 80%, which represents the ratio of non-zero parameters in the model; a higher second model sparsification degree indicates that more parameters in the denoising model are 0.

[0244] The sparsification degree of the first model and the sparsification degree of the second model may be the same or different.

[0245] S1502: Perform a pruning operation on the denoising model according to the sparsification degree of the second model and the parameter value of each parameter in the denoising model to obtain a pruned denoising model.

[0246] Get the parameter value of each parameter in the denoising model. The larger the parameter value, the more important the corresponding parameter is in the denoising model.

[0247] Therefore, according to the sparse ratio corresponding to the sparsification degree of the second model, the smaller parameter value can be determined as 0, thereby pruning the denoising model to obtain the pruned denoising model.

[0248] Specifically, according to the sparsity degree of the second model and the total number of parameters in the denoising model, the number of parameters that need to be pruned in the denoising model is determined, and the parameter values ​​of the parameters with smaller number of parameters are determined to be 0. For example, if the denoising model includes 10 parameters and the sparsity degree of the second model is 70%, the parameter values ​​of the smallest 3 parameters among the 10 parameters are determined to be 0.

[0249] S1503, quantizing the pruned denoising model to obtain a quantized denoising model; the quantized denoising model is used to predict a denoised video frame corresponding to the video frame.

[0250] The pruned denoising model is fine-tuned through multiple denoising training groups, and it is determined whether the model size, inference speed and accuracy of the fine-tuned denoising model meet the requirements. If not, the denoising model is pruned. If it meets the requirements, the fine-tuned denoising model is quantized to obtain a quantized denoising model.

[0251] Among them, model quantization is essentially to represent the high-bit weights and eigenvalues ​​in the model with lower bits; that is, the data format (32-bit double-precision floating point number) in the fine-tuned denoising model can be adjusted to 8-bit or 16-bit fixed-point number, and then the denoising model with the adjusted format is retrained through multiple denoising training groups (training perceptual quantization), so as to obtain a quantized denoising model. The quantized denoising model is a low-bit quantized inference model.

[0252] The video denoising method provided in the embodiment of the present application obtains the preset sparsity degree of the second model, and performs a pruning operation on the denoising model according to the sparsity degree of the second model and the parameter values ​​of each parameter in the denoising model to obtain a pruned denoising model, and then performs model quantization on the pruned denoising model to obtain a quantized denoising model; the quantized denoising model is used to predict the denoised video frame corresponding to the video frame. In this method, since quantizing and pruning the model can reduce the number of parameters in the model and reduce the computational complexity of the model, the denoised video frame of the video frame is predicted by the pruned and quantized denoising model, which improves the speed of predicting the denoised video frame, thereby improving the speed of video denoising.

[0253] The present application also provides a video denoising method, which includes a model training part and a model application part. In one embodiment, Fig.16 As shown, Fig.16 A method for model training is given, and the embodiment includes the following steps:

[0254] S1601, obtain training data pairs of GT data and LQ data in RGB format.

[0255] Among them, GT data is a number of continuous video frames that are clean and noise-free; LQ is a number of corresponding continuous video frames with noise.

[0256] S1602, performing block processing on the GT data and the LQ data to obtain multiple groups of GT data and LQ data.

[0257] S1603, sampling multiple groups of GT data and LQ data according to a preset number of frames, and performing data augmentation on the sampled data to obtain training data.

[0258] The sampling method may include continuous sampling or alternate frame sampling.

[0259] S1604, dividing the LQ data in the training data into multiple groups of training samples; each group of training samples includes a preset number of consecutive video frames, and the preset number of consecutive video frames includes a target frame and a reference frame.

[0260] S1605, obtaining an initial optical flow estimation model.

[0261] S1606, pruning and quantizing the initial optical flow estimation model to obtain an optical flow estimation model.

[0262] S1607, input each group of training samples into the optical flow estimation model respectively, obtain the optical flow field of each reference frame to the target frame, and align each reference frame to the target frame according to the optical flow field of each reference frame to the target frame to obtain the aligned reference frame and target frame.

[0263] S1608, connect the target frame and the reference frame through channels to obtain denoising model training data.

[0264] S1609, iteratively training the initial denoising model through the denoising model training data until the initial denoising model reaches a preset convergence condition, thereby obtaining a reference denoising model.

[0265] Among them, for the training of the initial denoising model with any denoising model training data, the denoising model training data is input into the initial denoising model to obtain the predicted noise-free result of the target frame; according to the predicted noise-free result of the target frame and the corresponding real GT data, the loss function value of the initial denoising model trained with the denoising model training data is determined, and the parameters of the initial denoising model are updated according to the loss function value until the initial denoising model reaches a preset convergence condition; the preset convergence condition can be that the number of iterations of the initial denoising model reaches a preset number threshold or the accuracy of the initial denoising model reaches a preset accuracy threshold. The initial denoising model can be a convolutional neural network (CNN).

[0266] S1610: Prune and quantize the reference denoising model to obtain a denoising model.

[0267] In one embodiment, Fig.17 As shown, Fig.17 The method of applying the model is given, and the embodiment includes the following steps:

[0268] S1701, obtaining an original video sequence captured by a shooting device.

[0269] S1702, performing image signal processing on the original video sequence to obtain the original video sequence in RGB format.

[0270] S1703, grouping the original video sequence in RGB format by a preset number of frames to obtain multiple groups of video sequences; each group of video sequences includes a video frame to be denoised and at least one reference video frame.

[0271] S1704: Obtain an optical flow field between each reference video frame and the video frame to be denoised according to the optical flow estimation model.

[0272] S1705 , aligning and registering each reference video frame with the video frame to be denoised according to the optical flow field between each reference video frame and the video frame to be denoised, to obtain aligned reference video frames and the video frame to be denoised.

[0273] S1706, inputting the aligned reference video frames and the video frames to be denoised into a denoising model, and outputting denoised video frames after denoising the video frames to be denoised through the denoising model.

[0274] S1707: Determine a denoised video sequence corresponding to the original video sequence according to the denoised video frames obtained after denoising the video frames to be denoised.

[0275] In an embodiment of the present application, in order to achieve real-time and high-quality denoising of high-definition video streams, an algorithm for joint temporal and spatial denoising is designed, and a high-quality data set construction method is proposed to construct clean and degraded noisy data pairs; and a fast optical flow estimation network is used to predict the optical flow field on a small-resolution image to pre-align the previous and next frames to the reference frame; supervised learning is performed on the CNN denoising network, and finally the performance of the two models is optimized through pruning and quantization, so that the entire method meets the real-time processing requirements of a low-power embedded platform.

[0276] Moreover, while the video noise is removed well, most of the details in the picture are retained, the picture is clean and natural without violent jumps and obvious ghosting, and it can be processed in real time on the embedded platform without frame drops.

[0277] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0278] Based on the same inventive concept, the embodiment of the present application also provides a video denoising device for implementing the above-mentioned video denoising method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more video denoising device embodiments provided below can refer to the limitations of the video denoising method above, and will not be repeated here.

[0279] In an exemplary embodiment, Fig.18 As shown, a video denoising device 1800 is provided, comprising: an acquisition module 1801, a compression module 1802 and a denoising module 1803, wherein:

[0280] An acquisition module 1801 is used to acquire an original video sequence;

[0281] The compression module 1802 is used to compress the original video sequence to obtain a compressed video sequence;

[0282] The denoising module 1803 is used to perform an optical flow alignment operation on the compressed video sequence according to a preset denoising model, and denoise the compressed video sequence after the optical flow alignment to obtain a denoised video sequence; the denoising model is obtained by pruning and quantizing the initial denoising model.

[0283] In one embodiment, the compression module 1802 includes:

[0284] The first conversion unit is used to perform a first format conversion on the original video sequence to obtain a compressed video sequence; the data volume of the compressed video sequence is smaller than the data volume of the original video sequence.

[0285] In one embodiment, the denoising module 1803 includes:

[0286] A first grouping unit is used to determine a plurality of groups of video sequences from the compressed video sequence; each group of video sequences includes any one video frame in the compressed video sequence and at least one reference frame corresponding to the video frame;

[0287] A denoising unit is used to perform an optical flow alignment operation on each group of video sequences according to a denoising model, and denoise each group of video sequences after the optical flow alignment, so as to obtain denoised video frames corresponding to the video frames in each group of video sequences;

[0288] The obtaining unit is used to obtain a denoised video sequence according to the denoised video frames corresponding to the video frames in each group of video sequences.

[0289] In one embodiment, the at least one reference frame includes a previous frame and a next frame of the video frame in the compressed video sequence.

[0290] In one embodiment, the denoising model includes an optical flow estimation model and a denoising model; the denoising unit includes:

[0291] A calculation subunit, for calculating the optical flow field between each reference frame and the video frame in the video sequence according to the optical flow estimation model for any group of video sequences;

[0292] A registration subunit, used for aligning each reference frame to the video frame according to the optical flow field to obtain aligned reference frames;

[0293] The denoising subunit is used to denoise the video frames according to the denoising model and the aligned reference frames to obtain denoised video frames corresponding to the video frames in the video sequence.

[0294] In one embodiment, the computing subunit includes:

[0295] A compression subunit, used for performing channel compression on the video frames and reference frames in the video sequence to obtain single-channel video frames and reference frames;

[0296] A first sampling subunit is used to downsample the single-channel video frame and each reference frame to obtain a sampled video frame and each reference frame;

[0297] The first obtaining subunit is used to input the sampled video frame and each reference frame into the optical flow estimation model to obtain the optical flow field between each reference frame in the video sequence and the video frame.

[0298] In one embodiment, the noise reduction subunit includes:

[0299] The first acquisition subunit is used to input the aligned reference frames and video frames into the noise reduction model to obtain a denoised video frame corresponding to the video frame.

[0300] In one embodiment, the obtaining unit comprises:

[0301] A conversion subunit, used for converting each denoised video frame into a second format to obtain a plurality of denoised video frames in the second format;

[0302] The first determining subunit is used to determine a denoised video sequence according to a plurality of denoised video frames in a second format.

[0303] In one embodiment, the apparatus 1800 further includes:

[0304] An initial model acquisition module is used to obtain an initial optical flow estimation model;

[0305] The optimization module is used to prune and quantize the initial optical flow estimation model to obtain the optical flow estimation model.

[0306] In one embodiment, the optimization module includes:

[0307] A sparse acquisition unit, used to acquire a preset first model sparsification degree;

[0308] A pruning unit, used for performing a pruning operation on the initial optical flow estimation model according to the sparsification degree of the first model and the parameter values ​​of each parameter in the initial optical flow estimation model to obtain a pruned initial optical flow estimation model;

[0309] The quantization unit is used to quantize the pruned initial optical flow estimation model to obtain a quantized initial optical flow estimation model; and determine the quantized initial optical flow estimation model as the optical flow estimation model.

[0310] In one embodiment, the apparatus 1800 further includes:

[0311] A training data acquisition module, used to acquire denoising training data; the denoising training data includes a plurality of denoising training groups; each denoising training group includes a target noisy video frame and at least one reference noisy video frame aligned with the target noisy video frame;

[0312] The training module is used to input each denoising training group into the initial denoising model respectively, and iteratively train the initial denoising model through each denoising training group until the initial denoising model converges to obtain the denoising model.

[0313] In one embodiment, the training data acquisition module includes:

[0314] A sample acquisition unit is used to acquire a noise reduction training sample; the noise reduction training sample includes a plurality of continuous noisy video frames and a corresponding plurality of continuous noise-free video frames;

[0315] A second grouping unit is used to group the noise reduction training samples to obtain a plurality of noise reduction sample groups; each noise reduction sample group includes a target noisy video frame and at least one reference noisy video frame;

[0316] An alignment unit, used for aligning each denoised sample group through an optical flow estimation model to obtain a plurality of aligned denoised sample groups;

[0317] The determination unit is used to determine the multiple aligned denoised sample groups as denoised training data.

[0318] In one embodiment, the sample acquisition unit includes:

[0319] The second acquisition subunit is used to acquire an original data set; the original data set includes a plurality of continuous noisy sample video frames and corresponding noise-free sample video frames;

[0320] The segmentation subunit is used to segment each sample video frame in the original data set to obtain the segmented original data set, and determine the noise reduction training sample according to the segmented original data set.

[0321] In one embodiment, the segmentation subunit comprises:

[0322] The augmentation subunit is used to perform data augmentation on the segmented original data set to obtain an augmented original data set;

[0323] The second determination subunit is used to determine the augmented original data set as the denoising training sample.

[0324] In one embodiment, the training module includes:

[0325] A training unit, used to train the initial denoising model through each denoising training group respectively, and obtain the training group loss function value of the initial denoising model trained by each denoising training group;

[0326] The updating unit is used to update the parameters of the initial denoising model according to the loss function values ​​of each training group until the initial denoising model meets the preset conditions and determines that the initial denoising model converges.

[0327] In one embodiment, the training unit includes:

[0328] A prediction subunit, for obtaining, for any denoising training group, a predicted denoised video frame of a target noisy video frame in the denoising training group through an initial denoising model;

[0329] The third determination subunit is used to determine the training group loss function value of the initial denoising model trained by the denoising training group according to the target noisy video frame and the predicted denoised video frame in the denoising training group.

[0330] In one embodiment, the apparatus 1800 further includes:

[0331] A sparse acquisition module, used to obtain a preset second model sparsification degree;

[0332] A pruning module, used for performing a pruning operation on the denoising model according to the sparsification degree of the second model and the parameter value of each parameter in the denoising model to obtain a pruned denoising model;

[0333] The quantization module is used to quantize the pruned denoising model to obtain a quantized denoising model; the quantized denoising model is used to predict a denoised video frame corresponding to the video frame.

[0334] Each module in the above-mentioned video denoising device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0335] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.19 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video denoising data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video denoising method is implemented.

[0336] Those skilled in the art will understand that Fig.19 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0337] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0338] The implementation principles and technical effects of each step implemented by the processor in the embodiment of the present application are similar to the principles of the above-mentioned video denoising method, and will not be repeated here.

[0339] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0340] The implementation principles and technical effects of the steps implemented when the computer program in the embodiment of the present application is executed by the processor are similar to the principles of the above-mentioned video denoising method, and will not be repeated here.

[0341] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0342] The implementation principles and technical effects of the steps implemented when the computer program in the embodiment of the present application is executed by the processor are similar to the principles of the above-mentioned video denoising method, and will not be repeated here.

[0343] It should be noted that the data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0344] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0345] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0346] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video denoising method, It is characterized in that The method comprises: Get the original video sequence; Performing data compression on the original video sequence to obtain a compressed video sequence; The compressed video sequence is subjected to an optical flow alignment operation according to a preset denoising model, and the compressed video sequence after the optical flow alignment is denoised to obtain a denoised video sequence; the denoising model is obtained by pruning and quantizing the initial denoising model.

2. The method according to claim 1, It is characterized in that The step of compressing the original video sequence to obtain a compressed video sequence comprises: The original video sequence is subjected to a first format conversion to obtain the compressed video sequence; the data volume of the compressed video sequence is smaller than the data volume of the original video sequence.

3. The method according to claim 1 or 2, It is characterized in that The step of performing an optical flow alignment operation on the compressed video sequence according to a preset denoising model, and denoising the compressed video sequence after the optical flow alignment to obtain a denoised video sequence comprises: Determine multiple groups of video sequences from the compressed video sequence; each group of the video sequences includes any one video frame in the compressed video sequence and at least one reference frame corresponding to the video frame; Performing optical flow alignment operations on each group of video sequences according to the denoising model, and denoising each group of video sequences after the optical flow alignment, to obtain denoised video frames corresponding to the video frames in each group of video sequences; The denoised video sequence is obtained according to the denoised video frames corresponding to the video frames in each group of video sequences.

4. The method according to claim 3, It is characterized in that The at least one reference frame includes a previous frame and a next frame of the video frame in the compressed video sequence.

5. The method according to claim 3, It is characterized in that The denoising model includes an optical flow estimation model and a denoising model; performing an optical flow alignment operation on each group of video sequences according to the denoising model, and denoising each group of video sequences after the optical flow alignment, to obtain denoised video frames corresponding to video frames in each group of video sequences, including: For any group of video sequences, calculating the optical flow field between each reference frame and the video frame in the video sequence according to the optical flow estimation model; According to the optical flow field, aligning each of the reference frames to the video frame to obtain aligned reference frames; The video frame is denoised according to the denoising model and the aligned reference frames to obtain a denoised video frame corresponding to the video frame in the video sequence.

6. The method according to claim 5, It is characterized in that The step of calculating the optical flow field between each reference frame and the video frame in the video sequence according to the optical flow estimation model includes: Channel-compressing the video frames and reference frames in the video sequence to obtain single-channel video frames and reference frames; Down-sampling the single-channel video frame and each reference frame to obtain a sampled video frame and each reference frame; The sampled video frames and each reference frame are input into the optical flow estimation model to obtain the optical flow field between each reference frame in the video sequence and the video frame.

7. The method according to claim 5, It is characterized in that The step of performing noise reduction on the video frame according to the noise reduction model and the aligned reference frames to obtain a denoised video frame corresponding to the video frame in the video sequence includes: The aligned reference frames and the video frame are input into the noise reduction model to obtain a denoised video frame corresponding to the video frame.

8. The method according to claim 3, It is characterized in that The step of obtaining the denoised video sequence according to the denoised video frames corresponding to the video frames in each group of video sequences comprises: Convert each of the denoised video frames into a second format to obtain a plurality of denoised video frames in the second format; The denoised video sequence is determined according to the plurality of denoised video frames in the second format.

9. The method according to claim 5, It is characterized in that The acquisition process of the optical flow estimation model includes: Get the initial optical flow estimation model; The initial optical flow estimation model is pruned and quantized to obtain the optical flow estimation model.

10. The method according to claim 9, It is characterized in that The pruning and quantizing of the initial optical flow estimation model to obtain the optical flow estimation model includes: Obtaining a preset first model sparsification degree; Performing a pruning operation on the initial optical flow estimation model according to the sparsification degree of the first model and the parameter value of each parameter in the initial optical flow estimation model to obtain a pruned initial optical flow estimation model; quantizing the pruned initial optical flow estimation model to obtain a quantized initial optical flow estimation model; The quantized initial optical flow estimation model is determined as the optical flow estimation model.

11. The method according to claim 5, It is characterized in that The training process of the denoising model includes: Acquire noise reduction training data; the noise reduction training data includes a plurality of noise reduction training groups; each noise reduction training group includes a target noisy video frame and at least one reference noisy video frame aligned with the target noisy video frame; The denoising training groups are input into the initial denoising model respectively, and the initial denoising model is iteratively trained by the denoising training groups until the initial denoising model converges, thereby obtaining the denoising model.

12. The method according to claim 11, It is characterized in that The obtaining of noise reduction training data comprises: Acquire a noise reduction training sample; the noise reduction training sample includes a plurality of consecutive noisy video frames and a corresponding plurality of consecutive noise-free video frames; The denoising training samples are grouped to obtain a plurality of denoising sample groups; each denoising sample group includes a target noisy video frame and at least one reference noisy video frame; Aligning the denoised sample groups by using the optical flow estimation model to obtain a plurality of aligned denoised sample groups; The plurality of aligned denoised sample groups are determined as the denoised training data.

13. The method according to claim 12, It is characterized in that The obtaining of the denoising training sample comprises: Acquire an original data set; the original data set includes a plurality of continuous noisy sample video frames and corresponding noise-free sample video frames; Each sample video frame in the original data set is segmented to obtain a segmented original data set, and the noise reduction training sample is determined according to the segmented original data set.

14. The method according to claim 13, It is characterized in that The step of determining the denoising training sample according to the segmented original data set includes: Performing data augmentation on the segmented original data set to obtain an augmented original data set; The augmented original data set is determined as the denoising training sample.

15. The method according to claim 11, It is characterized in that The iterative training of the initial denoising model by each of the denoising training groups until the initial denoising model converges includes: The initial denoising model is trained by each of the denoising training groups respectively, and a training group loss function value of the initial denoising model trained by each of the denoising training groups is obtained; The parameters of the initial denoising model are updated according to the loss function values ​​of each training group until the initial denoising model meets a preset condition, and it is determined that the initial denoising model converges.

16. The method according to claim 15, It is characterized in that The step of training the initial denoising model by each of the denoising training groups to obtain the training group loss function value of the initial denoising model trained by each of the denoising training groups includes: For any denoising training group, obtaining a predicted denoised video frame of a target noisy video frame in the denoising training group through the initial denoising model; According to the target noisy video frame and the predicted denoised video frame in the denoising training group, a training group loss function value of the initial denoising model trained by the denoising training group is determined.

17. The method according to claim 11, It is characterized in that After obtaining the noise reduction model, the method further includes: Obtaining a preset second model sparsification degree; According to the sparsification degree of the second model and the parameter value of each parameter in the denoising model, the denoising model is pruned to obtain a pruned denoising model; The pruned denoising model is quantized to obtain a quantized denoising model; the quantized denoising model is used to predict a denoised video frame corresponding to the video frame.

18. A video denoising device, It is characterized in that The device comprises: An acquisition module, used for acquiring an original video sequence; A compression module, used for compressing the original video sequence to obtain a compressed video sequence; The denoising module is used to perform an optical flow alignment operation on the compressed video sequence according to a preset denoising model, and denoise the compressed video sequence after the optical flow alignment to obtain a denoised video sequence; the denoising model is obtained by pruning and quantizing the initial denoising model.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 17 are implemented.

20. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

21. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images