Video denoising method, device and terminal

Abstract

The present invention provides a video denoising method, device and terminal. The method comprises: sequentially acquiring an image sequence in a video; denoising the first frame of the image in the image sequence; for the Nth frame of the image after the first frame of the image, referring to the denoised image of the N-1th frame, utilizing the correlation between the Nth frame of the image and the denoised image of the N-1th frame, adopting a denoising model based on deep learning to perform denoising, and completing the denoising of all images in the image sequence. The present invention can more fully utilize the temporal information of the video in the denoising process and optimize the denoising effect by screening the information of the previous frame that is helpful for the denoising of the next frame.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video denoising method, device and terminal. Background Art

[0002] Usually, there is noise in the video. The visibility of the noisy video is poor, which will affect people's perception to a certain extent. In addition, it will also affect the efficiency of video encoding and the performance of computer vision algorithms that process the video. Therefore, before encoding the video or performing computer vision processing based on the video, denoising the video is an essential process.

[0003] Among the existing video denoising methods, one method is to use the information of the previous frame image in the video to denoise the current frame image. However, this method uses a motion detection module to detect which areas belong to the static area and which areas belong to the moving area. The next frame image only uses the information of the static area of ​​the previous frame image, and the moving area uses the ordinary denoising method. This method does not make full use of the useful information of the previous frame image, and the denoising effect is not good enough; it also requires the design of an additional motion detection module, which is more complicated to implement. Summary of the invention

[0004] The video denoising method, device and terminal provided by the present invention can more fully utilize the timing information of the video in the denoising process and optimize the denoising effect by screening the information of the previous frame that is helpful for the denoising of the next frame.

[0005] In a first aspect, the present invention provides a video denoising method, the method comprising:

[0006] Sequentially acquire the image sequence in the video;

[0007] Performing denoising processing on the first frame image in the image sequence;

[0008] For the Nth frame image after the first frame image, refer to the denoised image of the N-1th frame, utilize the correlation between the Nth frame image and the denoised image of the N-1th frame, adopt a denoising model based on deep learning to perform denoising, and complete the denoising processing of all images in the image sequence.

[0009] Optionally, the step of performing denoising on the Nth frame image after the first frame image with reference to the denoised image of the N-1th frame, utilizing the correlation between the Nth frame image and the denoised image of the N-1th frame, and adopting a denoising model based on deep learning includes:

[0010] A feature map is extracted from the denoised image of the N-1th frame by the second convolution combination module, a feature map is extracted from the non-denoised image of the Nth frame by the third convolution combination module, and the two feature maps are spliced ​​by the splicing module;

[0011] The above concatenated feature maps are used by the attention module to calculate a weight matrix in the range of [0, 1];

[0012] The denoised image of the N-1th frame is extracted into a feature map through the first convolution combination module, and the feature map is weighted multiplied with the weight matrix through the multiplication module to obtain a weighted feature map;

[0013] The Nth frame of non-denoised image is passed through the fourth convolution combination module CNN_Block4 to extract the feature map, and then it is spliced ​​with the above weighted feature map through the splicing module;

[0014] The above-mentioned spliced ​​feature map is passed through the fifth convolution combination module to calculate the noise of the Nth frame image.

[0015] The noise of the Nth frame image calculated above is subtracted from the Nth frame non-denoised image by a subtraction module to obtain the Nth frame denoised image.

[0016] Optionally, the convolution combination module is any combination of multiple convolutions, activation functions, normalization and residual operators.

[0017] Optionally, the splicing module is used to splice two feature maps into one feature map, that is, to splice a feature map with a dimension of (n, c1, h, w) and a feature map with a dimension of (n, c2, h, w) into a feature map with a dimension of (n, c1+c2, h, w), where n is the number of input samples, c1 and c2 are the number of channels of the feature map, h is the height of the feature map, and w is the width of the feature map.

[0018] In a second aspect, the present invention provides a video denoising device, the device comprising:

[0019] An acquisition unit, used for sequentially acquiring an image sequence in a video;

[0020] A first denoising unit, used for performing denoising processing on a first frame of image in the image sequence;

[0021] The second denoising unit is used to perform denoising on the Nth frame image after the first frame image, with reference to the denoised image of the N-1th frame, utilizing the correlation between the Nth frame image and the denoised image of the N-1th frame, and adopting a denoising model based on deep learning to complete the denoising processing of all images in the image sequence.

[0022] Optionally, the second denoising unit includes:

[0023] The second convolution combination module is used to extract a feature map from the denoised image of the N-1th frame;

[0024] The third convolution combination module is used to extract a feature map from the Nth frame of the non-denoised image;

[0025] A first splicing module, used for splicing the second feature map extracted by the second convolution combination module and the feature map extracted by the third convolution combination module;

[0026] An attention module, used to calculate a weight matrix in the range of [0, 1] for the feature map obtained after the first splicing module is spliced;

[0027] The first convolution combination module is used to extract a feature map from the N-1th frame of denoised image;

[0028] A multiplication module, used for performing weighted multiplication on the feature map extracted by the first convolution combination module and the weight matrix to obtain a weighted feature map;

[0029] The fourth convolution combination module is used to extract a feature map from the N-th frame of the non-denoised image;

[0030] A second splicing module, used to splice the feature map extracted by the fourth convolution combination module with the weighted feature map;

[0031] A fifth convolution combination module, used to calculate the noise of the Nth frame image based on the feature map obtained after the second splicing module splices;

[0032] The subtraction module is used to subtract the noise of the Nth frame image from the Nth frame non-denoised image to obtain the Nth frame denoised image.

[0033] Optionally, the convolution combination module is any combination of multiple convolutions, activation functions, normalization and residual operators.

[0034] Optionally, the splicing module is used to splice two feature maps into one feature map, that is, to splice a feature map with a dimension of (n, c1, h, w) and a feature map with a dimension of (n, c2, h, w) into a feature map with a dimension of (n, c1+c2, h, w), where n is the number of input samples, c1 and c2 are the number of channels of the feature map, h is the height of the feature map, and w is the width of the feature map.

[0035] In a third aspect, the present invention provides a terminal, the terminal comprising:

[0036] at least one processor; and

[0037] a memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned video denoising method.

[0039] In a fourth aspect, the present invention provides a chip, comprising:

[0040] at least one processor; and

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned video denoising method.

[0043] In a fifth aspect, the present invention provides a chip module, wherein the chip module comprises the above-mentioned chip.

[0044] In a sixth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions implement the above-mentioned video denoising method when executed by a processor.

[0045] The video denoising method, device and terminal provided by the embodiments of the present invention use a deep learning-based denoising model whose input includes not only the N-th frame of non-denoised image but also the N-1-th frame of denoised image. In this way, the similarity between the two frames of images can be utilized, and by screening the information of the previous frame that is helpful for denoising the next frame, the timing information of the video can be more fully utilized in the denoising process, thereby optimizing the denoising effect of the N-1-th frame of image. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a video denoising method according to an embodiment of the present invention;

[0047] Figure 2 A flowchart of a video denoising method according to another embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the structure of a denoising model based on deep learning provided in an embodiment of the present invention;

[0049] Figure 4 It is a structural schematic diagram of a video denoising device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The embodiment of the present invention provides a video denoising method, which is applied to a terminal, such as Figure 1 As shown, the method includes:

[0052] S11, sequentially acquiring a sequence of images in the video;

[0053] S12, performing denoising processing on the first frame image in the image sequence;

[0054] S13. For the Nth frame image after the first frame image, refer to the denoised image of the N-1th frame, utilize the correlation between the Nth frame image and the denoised image of the N-1th frame, adopt a denoising model based on deep learning to perform denoising, and complete the denoising processing of all images in the image sequence.

[0055] The video denoising method provided by an embodiment of the present invention adopts a deep learning-based denoising model whose input includes, in addition to the non-denoised image of the Nth frame, the denoised image of the N-1th frame. In this way, the similarity between the two frames of images can be utilized, and by screening the information of the previous frame that is helpful for denoising the next frame, the timing information of the video can be more fully utilized in the denoising process, thereby optimizing the denoising effect of the Nth frame image with the denoising effect of the N-1th frame image.

[0056] The video denoising method of the present invention is described in detail below with reference to specific embodiments.

[0057] like Figure 2 As shown, the video denoising method provided in this embodiment includes:

[0058] S21. Sequentially obtain a sequence of images in the video.

[0059] S22: performing denoising processing on the first frame image in the image sequence.

[0060] Specifically, a conventional image denoising method can be used to denoise the first frame image, wherein the conventional image denoising method can be a traditional denoising method, such as Gaussian filtering, guided filtering, Non-Local Mean, etc., or it can be an image denoising method based on deep learning.

[0061] S23. For the Nth frame image after the first frame image, refer to the denoised image of the N-1th frame, utilize the correlation between the Nth frame image and the denoised image of the N-1th frame, and adopt a denoising model based on deep learning to perform denoising to obtain the denoised Nth frame image.

[0062] The input of the deep learning-based denoising model includes not only the non-denoised image of the Nth frame, but also the denoised image of the N-1th frame. In this way, the similarity between the two frames of images can be utilized to optimize the denoising effect of the Nth frame with the denoising effect of the N-1th frame image.

[0063] The structure of the deep learning-based denoising model is as follows: Figure 3 As shown, the specific process is as follows:

[0064] The denoised image of the N-1th frame is passed through the second convolution combination module CNN_Blocks2 to extract the feature map, and the non-denoised image of the Nth frame is passed through the third convolution combination module CNN_Block3 to extract the feature map, and the two feature maps are spliced ​​through the splicing module;

[0065] The above concatenated feature maps are passed through the attention module to calculate the weight matrix in the range of [0, 1];

[0066] The N-1th frame of denoised image is extracted from the first convolution combination module CNN_Blocks1 to obtain a feature map, and the feature map is weighted multiplied with the weight matrix through the multiplication module to obtain a weighted feature map to prompt the neural network to focus on which parts of the feature map of the N-1th frame of denoised image. Parts with high similarity and high correlation with the Nth frame of non-denoised image will receive higher weights during the learning process;

[0067] The Nth frame of non-denoised image is passed through the fourth convolution combination module CNN_Block4 to extract the feature map, and then it is spliced ​​with the above weighted feature map through the splicing module;

[0068] The above-mentioned spliced ​​feature map is passed through the fifth convolution combination module CNN_Blocks5 to calculate the noise of the Nth frame image;

[0069] The noise of the Nth frame image calculated above is subtracted from the Nth frame non-denoised image by a subtraction module to obtain the Nth frame denoised image.

[0070] The functions implemented by each module are as follows:

[0071] Convolution combination module: It can be a combination of multiple convolution + activation function + normalization + residual operators, and the combination method can be freely designed. Optionally, it can be a series combination of n 3×3 convolutions + RELU.

[0072] Splicing module: Splice two feature maps into one feature map, that is, splice the feature map with dimensions (n, c1, h, w) and the feature map with dimensions (n, c2, h, w) into a feature map with dimensions (n, c1 + c2, h, w). Among them, n is the number of input samples, c1 and c2 are the number of channels of the feature map, h is the height of the feature map, and w is the width of the feature map.

[0073] Attention module: This module is a combination of multiple convolution and activation functions. The output of this module can be activated by sigmoid and other activation functions to limit the output to [0, 1].

[0074] Multiplication module: Multiply the values ​​of corresponding positions of two feature maps.

[0075] Subtraction module: subtract the values ​​of corresponding positions of two feature maps.

[0076] The parameters of the above denoising model are obtained after learning a large number of training samples. The training samples are a large number of consecutive two-frame image groups. The input samples are the previous clean image and the next noisy image. The supervision sample is the next clean image. The loss function can use any one or a combination of losses such as L1, L2, SSIM (Structural Similarity). The learning method can use any deep learning optimization method such as gradient descent, ADAM (Adaptive Moment Estimation, adaptive moment estimation).

[0077] S24, caching the Nth frame image after denoising.

[0078] Steps S23 and S24 are sequentially performed on each frame of image after the first frame of image until all images are denoised. At this point, the video denoising is completed.

[0079] The video denoising method provided by an embodiment of the present invention adopts a deep learning-based denoising model whose input includes, in addition to the non-denoised image of the Nth frame, the denoised image of the N-1th frame. In this way, the similarity between the two frames of images can be utilized, and the attention neural network can be used to filter the information of the previous frame that is helpful for denoising the next frame. In the denoising process, the timing information of the video is more fully utilized, so that the denoising effect of the N-1th frame image can be used to optimize the denoising effect of the Nth frame image.

[0080] The embodiment of the present invention also provides a video denoising device, the device is located in a terminal, such as Figure 4 As shown, the device comprises:

[0081] An acquisition unit 11 is used to sequentially acquire an image sequence in a video;

[0082] A first denoising unit 12, configured to perform denoising processing on a first frame of image in the image sequence;

[0083] The second denoising unit 13 is used to perform denoising on the Nth frame image after the first frame image, with reference to the denoised image of the N-1th frame, utilizing the correlation between the Nth frame image and the denoised image of the N-1th frame, and adopting a denoising model based on deep learning to complete the denoising processing of all images in the image sequence.

[0084] The video denoising device provided by an embodiment of the present invention adopts a deep learning-based denoising model whose input includes, in addition to the non-denoised image of the Nth frame, the denoised image of the N-1th frame. In this way, the similarity between the two frames of images can be utilized, and by screening the information of the previous frame that is helpful for denoising the next frame, the timing information of the video can be more fully utilized in the denoising process, thereby optimizing the denoising effect of the Nth frame image with the denoising effect of the N-1th frame image.

[0085] Optionally, the second denoising unit includes:

[0086] The second convolution combination module is used to extract a feature map from the denoised image of the N-1th frame;

[0087] The third convolution combination module is used to extract a feature map from the Nth frame of the non-denoised image;

[0088] A first splicing module, used for splicing the second feature map extracted by the second convolution combination module and the feature map extracted by the third convolution combination module;

[0089] An attention module, used to calculate a weight matrix in the range of [0, 1] for the feature map obtained after the first splicing module is spliced;

[0090] The first convolution combination module is used to extract a feature map from the N-1th frame of denoised image;

[0091] A multiplication module, used for performing weighted multiplication on the feature map extracted by the first convolution combination module and the weight matrix to obtain a weighted feature map;

[0092] The fourth convolution combination module is used to extract a feature map from the N-th frame of the non-denoised image;

[0093] A second splicing module, used to splice the feature map extracted by the fourth convolution combination module with the weighted feature map;

[0094] A fifth convolution combination module, used to calculate the noise of the Nth frame image based on the feature map obtained after the second splicing module splices;

[0095] The subtraction module is used to subtract the noise of the Nth frame image from the Nth frame non-denoised image to obtain the Nth frame denoised image.

[0096] Optionally, the convolution combination module is any combination of multiple convolutions, activation functions, normalization and residual operators.

[0097] Optionally, the splicing module is used to splice two feature maps into one feature map, that is, to splice a feature map with a dimension of (n, c1, h, w) and a feature map with a dimension of (n, c2, h, w) into a feature map with a dimension of (n, c1+c2, h, w), where n is the number of input samples, c1 and c2 are the number of channels of the feature map, h is the height of the feature map, and w is the width of the feature map.

[0098] The device of this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0099] An embodiment of the present invention further provides a terminal, comprising:

[0100] at least one processor; and

[0101] a memory communicatively connected to the at least one processor; wherein,

[0102] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned video denoising method.

[0103] An embodiment of the present invention further provides a chip, comprising:

[0104] at least one processor; and

[0105] a memory communicatively connected to the at least one processor; wherein,

[0106] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned video denoising method.

[0107] The present invention provides a chip module, which includes the above chip.

[0108] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions implement the above-mentioned video denoising method when executed by a processor.

[0109] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above method embodiments. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0110] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A video denoising method, characterized in that: The method comprises: Sequentially acquire the image sequence in the video; Performing denoising processing on the first frame image in the image sequence; For the Nth frame image after the first frame image, with reference to the denoised image of the N-1th frame, using the correlation between the Nth frame image and the denoised image of the N-1th frame, adopting a denoising model based on deep learning to perform denoising, and completing denoising processing for all images in the image sequence; The step of performing denoising on the Nth frame image after the first frame image with reference to the denoised image of the N-1th frame, utilizing the correlation between the Nth frame image and the denoised image of the N-1th frame, and adopting a denoising model based on deep learning includes: A feature map is extracted from the denoised image of the N-1th frame by the second convolution combination module, a feature map is extracted from the non-denoised image of the Nth frame by the third convolution combination module, and the two feature maps are spliced ​​by the splicing module; The above concatenated feature maps are used by the attention module to calculate a weight matrix in the range of [0, 1]; The denoised image of the N-1th frame is extracted into a feature map through the first convolution combination module, and the feature map is weighted multiplied with the weight matrix through the multiplication module to obtain a weighted feature map; The Nth frame of non-denoised image is passed through the fourth convolution combination module CNN_Block4 to extract the feature map, and then it is spliced ​​with the above weighted feature map through the splicing module; The above-mentioned spliced ​​feature map passes through the fifth convolution combination module to calculate the noise of the Nth frame image; The noise of the Nth frame image calculated above is subtracted from the Nth frame non-denoised image by a subtraction module to obtain the Nth frame denoised image.

2. The method according to claim 1, characterized in that The convolution combination module is any combination of multiple convolutions, activation functions, normalization and residual operators.

3. The method according to claim 1, characterized in that The splicing module is used to splice two feature maps into one feature map, that is, to splice a feature map with a dimension of (n, c1, h, w) and a feature map with a dimension of (n, c2, h, w) into a feature map with a dimension of (n, c1+c2, h, w), where n is the number of input samples, c1 and c2 are the number of channels of the feature map, h is the height of the feature map, and w is the width of the feature map.

4. A video denoising device, characterized in that: The device comprises: An acquisition unit, used for sequentially acquiring an image sequence in a video; A first denoising unit, used for performing denoising processing on a first frame of image in the image sequence; A second denoising unit is used to perform denoising on the Nth frame image after the first frame image with reference to the denoised image of the N-1th frame, utilize the correlation between the Nth frame image and the denoised image of the N-1th frame, and adopt a denoising model based on deep learning to complete the denoising process of all images in the image sequence; The second denoising unit comprises: The second convolution combination module is used to extract a feature map from the denoised image of the N-1th frame; The third convolution combination module is used to extract a feature map from the Nth frame of the non-denoised image; A first splicing module, used for splicing the second feature map extracted by the second convolution combination module and the feature map extracted by the third convolution combination module; An attention module, used to calculate a weight matrix in the range of [0, 1] for the feature map obtained after the first splicing module is spliced; The first convolution combination module is used to extract a feature map from the N-1th frame of denoised image; A multiplication module, used for performing weighted multiplication on the feature map extracted by the first convolution combination module and the weight matrix to obtain a weighted feature map; The fourth convolution combination module is used to extract a feature map from the N-th frame of the non-denoised image; A second splicing module, used to splice the feature map extracted by the fourth convolution combination module with the weighted feature map; A fifth convolution combination module, used to calculate the noise of the Nth frame image based on the feature map obtained after the second splicing module splices; The subtraction module is used to subtract the noise of the Nth frame image from the Nth frame non-denoised image to obtain the Nth frame denoised image.

5. The device according to claim 4, characterized in that The convolution combination module is any combination of multiple convolutions, activation functions, normalization and residual operators.

6. The device according to claim 4, characterized in that The splicing module is used to splice two feature maps into one feature map, that is, to splice a feature map with a dimension of (n, c1, h, w) and a feature map with a dimension of (n, c2, h, w) into a feature map with a dimension of (n, c1+c2, h, w), where n is the number of input samples, c1 and c2 are the number of channels of the feature map, h is the height of the feature map, and w is the width of the feature map.

7. A terminal, characterized in that: The terminal comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.

8. A chip, characterized in that: The chip comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.

9. A chip module, characterized in that: The chip module comprises the chip according to claim 8.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images