Video noise reduction method and device, electronic equipment and storage medium

Abstract

The invention discloses a video denoising method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-denoised first image and a denoised second image in a to-be-denoised video, the first image being any one video frame image in the to-be-denoised video, the second image being a previous frame image of the first image, and the first image being any one video frame image in the to-be-denoised video; the size of the second image is not greater than that of the first image, performing alignment processing on the first image and the second image to obtain an alignment image corresponding to the second image, performing multi-scale noise reduction processing according to the first image and the alignment image to obtain a target image after noise reduction of the first image, and replacing the first image in the video to be subjected to noise reduction with the target image. By adopting the embodiment of the invention, the problem of ghosting caused by noise reduction of multiple frames of videos can be relieved and avoided, and the noise reduction effect of the videos is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video noise reduction method, device, electronic equipment and storage medium. Background Art

[0002] Image noise is a random variation in brightness or color information in an image (not present in the object itself), usually a manifestation of electronic noise. It is generally generated by the sensors and circuits of scanners or digital cameras, or by film grain or the inevitable shot noise in ideal photodetectors. Image noise is an undesirable byproduct of the image capture process, which adds errors and extra information to the image.

[0003] Video is composed of multiple continuous video frames, and one video frame is an image, so video denoising includes denoising the images in the video frame. Image denoising refers to the technology of using certain methods to suppress or eliminate noise in the image while maintaining the original texture and details of the image as much as possible to improve the visual quality of the image. The difference between video denoising and single-frame image denoising is that video can use the correlation between video frames in the time domain and / or spatial domain.

[0004] In the related art, there are two main schemes for processing video noise reduction: one is a single-frame scheme, in which each frame is independently denoised through various methods and is not associated with the information of other frames; the other is a multi-frame scheme, which usually combines the noise reduction result of the previous frame with the current frame in various ways to obtain the noise reduction result of the current frame, and the previous frame and the current frame have the same resolution.

[0005] However, the single-frame solution does not consider the information in the time domain, resulting in noise flickering in the denoised video, which seriously affects the denoising effect; the multi-frame solution is prone to ghosting if there is motion in the shooting scene due to the inconsistent shooting time of the previous and next frames, which affects the denoising effect. Therefore, the video denoising solutions provided in the related art cannot effectively improve the denoising effect. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide a video noise reduction method, device, electronic device and storage medium to solve the technical problem that the video noise reduction solution provided in the related art cannot effectively improve the noise reduction effect.

[0007] In a first aspect, an embodiment of the present invention provides a video noise reduction method, comprising:

[0008] Acquire a first image to be denoised and a second image after denoising in the video to be denoised, wherein the first image is any frame image of the video to be denoised, the second image is a frame image previous to the first image, and the size of the second image is not greater than the size of the first image;

[0009] Performing alignment processing on the first image and the second image to obtain an aligned image corresponding to the second image;

[0010] Multi-scale noise reduction processing is performed according to the first image and the aligned image to obtain a target image after the noise reduction of the first image, and the target image replaces the first image in the video to be denoised.

[0011] In a second aspect, an embodiment of the present invention provides a video noise reduction device, including:

[0012] An acquisition module, used to acquire a first image to be denoised and a second image after denoising in the video to be denoised, wherein the first image is any frame image of the video to be denoised, and the second image is a frame image previous to the first image;

[0013] an alignment module, configured to perform alignment processing on the first image and the second image to obtain an aligned image corresponding to the second image;

[0014] The first denoising module is used to perform multi-scale denoising processing on the first image and the aligned image to obtain a target image after denoising the first image, and replace the first image in the video to be denoised with the target image.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned video noise reduction methods when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any one of the above-mentioned video noise reduction methods are implemented.

[0017] An embodiment of the present invention provides a video denoising method, device, electronic device and storage medium. The method obtains a first image to be denoised of any frame in the video to be denoised and a second image that is a frame before the first image and has been denoised, wherein the size of the second image is not larger than the size of the first image, and then performs multi-scale denoising processing based on the first image and the aligned image, so as to perform denoising processing on the first image based on the image information of the denoised second image. This method can alleviate and avoid the ghosting problem caused by multi-frame video denoising, and effectively improve the denoising effect on the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a video noise reduction method provided by an embodiment of the present invention;

[0019] Figure 2 Schematic diagram of a network structure of a multi-scale denoising model provided by an embodiment of the present invention;

[0020] Figure 3 is another flow chart of a video noise reduction method provided by an embodiment of the present invention;

[0021] Figure 4 is a structural schematic diagram of a video noise reduction device provided by an embodiment of the present invention;

[0022] Figure 5 is another structural schematic diagram of a video noise reduction device provided by an embodiment of the present invention;

[0023] Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present invention;

[0024] Figure 7 This is another structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0028] In the related art, there are two main schemes for processing video noise reduction: one is a single-frame scheme, in which each frame is independently denoised through various methods and is not associated with the information of other frames; the other is a multi-frame scheme, which usually combines the noise reduction result of the previous frame with the current frame in various ways to obtain the noise reduction result of the current frame, and the previous frame and the current frame have the same resolution.

[0029] However, the single-frame solution does not consider the information in the time domain, resulting in noise flickering in the denoised video, which seriously affects the denoising effect; the multi-frame solution is prone to ghosting if there is motion in the shooting scene due to the inconsistent shooting time of the previous and next frames, which affects the denoising effect. Therefore, the video denoising solutions provided in the related art cannot effectively improve the denoising effect.

[0030] In order to solve the technical problems existing in the related art, an embodiment of the present invention provides a video noise reduction method, which can be applied to electronic devices. When recording an object through a camera of an electronic device, due to limitations such as hardware and environment, the acquired video may have noise problems such as noise points, patches, ghosting, etc. that affect the visual effect. At this time, the original video without noise reduction collected by the camera in the electronic device can be subjected to noise reduction processing to obtain a video after noise reduction.

[0031] Among them, the above-mentioned electronic device can be mobile or fixed. For example, the electronic device can be a mobile phone with video noise reduction function, a camera, a camcorder, a vehicle, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a smart watch, augmented reality (AR) / virtual reality (VR), a wearable device (WD), a game console, etc. The embodiment of the present application does not limit the specific type of the electronic device.

[0032] For details, see Figure 1 , Figure 11 is a flow chart of a video noise reduction method provided by an embodiment of the present invention, the method comprising steps 101 to 103;

[0033] Step 101, obtaining a first image to be denoised and a second image after denoising in the video to be denoised, wherein the first image is any frame image of the video to be denoised, and the second image is a frame image before the first image, and the size of the second image is not greater than the size of the first image.

[0034] In this embodiment, the second image provided in this embodiment is an image after noise reduction processing. The specific noise reduction method can be to use a traditional noise reduction algorithm or an artificial intelligence noise reduction algorithm of a trained convolutional neural network for processing, which is not specifically limited here.

[0035] Among them, the first image to be denoised provided in this embodiment can be an RGB three-channel color image that has undergone image signal processing (ImageSignal Process, ISP), or it can be an original RAW image that has not been processed by ISP. In the case where the first image to be denoised is a three-channel color image that has been processed by ISP, the video denoising method provided in this embodiment can be used as a post-processing solution to optimize the video that the user has already shot, and improve the noise reduction effect and inter-frame consistency. When the first image to be denoised is an original RAW image that has not been processed by ISP, the video denoising method provided in this embodiment can be used as a noise reduction module in the mobile video ISP to perform noise reduction on the original RAW image in real time.

[0036] Step 102: align the first image and the second image to obtain an aligned image corresponding to the second image.

[0037] In this embodiment, before the step of aligning the first image and the second image to obtain an aligned image corresponding to the second image provided in this embodiment, the video noise reduction method provided in this embodiment may also include: transforming the first image and the second image to obtain a first transformed image and a second transformed image of the same size.

[0038] Among them, the transformation processing provided in this embodiment is mainly to transform the first image and / or the second image. Specifically, the transformation processing provided in this embodiment includes but is not limited to size transformation, color space transformation, downsampling and other processing. As long as the processing efficiency of the alignment processing of the first image and the second image can be improved, no specific limitation is made here.

[0039] Specifically, after the first image and the second image are transformed to obtain the first transformed image and the second transformed image of the same size, the step of aligning the first image and the second image provided in this embodiment to obtain the aligned image corresponding to the second image may be: aligning the first transformed image and the second transformed image to obtain the aligned image corresponding to the second transformed image. By using the first transformed image and the second transformed image of the same size for alignment processing, the processing efficiency of the alignment processing can be effectively improved. At the same time, by aligning the denoised second transformed image to the first transformed image, it is possible to facilitate the subsequent introduction of the denoised information in the denoised second transformed image, thereby reducing the ghosting problem caused by the misalignment of pixels due to the motion of the previous and next frames.

[0040] Optionally, since the size of the second image provided in this embodiment is not larger than the size of the first image, the size of the second image provided in this embodiment may be smaller than the size of the first image, and the size of the second image may also be equal to the size of the first image. In some embodiments, when the size of the second image is smaller than the size of the first image, it is necessary to first unify the size between the first image and the second image to reduce the size of the first image to the size of the second image, and then perform the alignment process. In this case, since the size of the second image after denoising is smaller and the resolution is lower, the amount of calculation when the first image and the second image are aligned can be effectively reduced. In other embodiments, when the size of the second image is equal to the size of the first image, the alignment process can be performed directly. In this case, since the size and resolution of the second image after denoising are not reduced, the alignment process is directly performed on the first image and the second image, which can effectively utilize the denoised information in the high-resolution second image and improve the denoising effect of the subsequent denoising process on the first image.

[0041] In some embodiments, the present embodiment may also transform the first image and the second image at the same time. Optionally, the transform process provided by the present embodiment may include downsampling. Then, the step of transforming the first image and the second image provided by the present embodiment to obtain the first transformed image and the second transformed image of the same size may be: downsampling the first image and the second image of the first size to obtain the first transformed image and the second transformed image of the second size, wherein the second size is not greater than the first size. By downsampling the first image and the second image at the same time, the amount of calculation when the first image and the second image are aligned can be effectively reduced, thereby improving the noise reduction efficiency of the video noise reduction.

[0042] It should be noted that the transformation process provided in this embodiment may also include upsampling processing, that is, the first image and the second image may be upsampled at the same time to obtain the first changed image and the second transformed image of the same third size, where the third size is larger than the first size. In this way, the denoised information in the high-resolution second transformed image may be effectively utilized to further improve the denoising effect of the subsequent denoising process on the first image.

[0043] In addition, the transformation processing provided in this embodiment can not only transform the size of the image, but also transform and select the color space. For example, when the first image to be denoised and the second image after denoising are three-channel color images, and the brightness channel image of the first image to be denoised and the brightness channel image of the second image after denoising are involved in the alignment processing, the aligned image finally input into the multi-scale denoising model is also the brightness image after the alignment processing. In this way, by using only a single-channel brightness map to participate in the calculation, the amount of calculation required for the denoising process can be further reduced while improving the denoising effect, which is beneficial to the deployment of mobile terminals.

[0044] As an optional embodiment, in order to reduce the interference of the noise existing in the first image on the alignment process to improve the alignment accuracy. This embodiment can perform noise reduction processing on the first image before performing the alignment processing on the first image and the second image to reduce the noise existing in the first image. Specifically, before the step of aligning the first transformed image and the second transformed image to obtain the aligned image corresponding to the second transformed image, the video noise reduction method provided by this embodiment can also include: performing noise reduction processing on the first transformed image to obtain a first noise-reduced image; performing motion estimation processing based on the first noise-reduced image and the second transformed image to obtain a motion estimation of the second transformed image relative to the first noise-reduced image.

[0045] In the process of performing denoising on the first transformed image, this embodiment may use a low-precision denoising algorithm to perform denoising on the first transformed image, thereby reducing the amount of calculation in the denoising process; or may use a high-precision denoising algorithm to perform denoising on the first transformed image, thereby improving the estimation accuracy of subsequent motion estimation and effectively reducing the interference of noise in the first image on the alignment process. Specifically, the accuracy of the denoising algorithm used by this embodiment to perform denoising on the first transformed image may be set according to actual application requirements and is not specifically limited here.

[0046] In this embodiment, after obtaining the motion estimation of the first denoised image and the second transformed image relative to the first denoised image, the step of aligning the first transformed image and the second transformed image to obtain the aligned image corresponding to the second transformed image provided in this embodiment may be: aligning the first denoised image and the second transformed image according to the motion estimation to obtain the aligned image corresponding to the second transformed image. In this embodiment, by aligning the second transformed image to the first denoised image, it is possible to reduce the ghosting problem caused by the misalignment of pixels due to the motion between the previous and next frames when subsequently introducing the denoised information of the previous frame.

[0047] The alignment process provided in this embodiment may include operations such as rotation and translation. The motion estimation process provided in this embodiment mainly estimates the motion of the second transformed image relative to the first denoised image, thereby generating corresponding alignment parameters, which facilitates the control of the alignment process of the first denoised image and the second transformed image, and effectively improves the alignment accuracy of the alignment process.

[0048] Step 103: Perform multi-scale noise reduction processing on the first image and the aligned image to obtain a target image after noise reduction of the first image, and replace the first image in the video to be noise reduced with the target image.

[0049] In this embodiment, by performing multi-scale noise reduction processing on the first image to be denoised and the aligned image of different scales, the embodiment can improve the consistency of the noise reduction effect of the previous and next frames, and effectively reduce the ghost problem caused by the movement of the previous and next frames, thereby achieving the purpose of improving the noise reduction effect of the video noise reduction.

[0050] Optionally, the present embodiment may use a pre-trained multi-scale denoising model to perform multi-scale denoising on the first image and the aligned image to obtain a target image after the first image is denoised. Specifically, the step of performing multi-scale denoising on the first image and the aligned image to obtain a target image after the first image is denoised provided in the present embodiment may be: inputting the first image and the aligned image into a pre-trained multi-scale denoising model to obtain a target image after the first image is denoised.

[0051] Specifically, before the step of performing multi-scale denoising processing according to the first image and the aligned image to obtain the target image after denoising the first image provided in the present embodiment, the video denoising method provided in the present embodiment may further include: obtaining a training data set and a multi-scale denoising model to be trained, the training data set including a first training image of the first frame in the training video that has been denoised, a second training image of any frame that has not been denoised, and an annotated image of the second training image that has been denoised, and the first training image including training images of multiple different sizes; inputting the first training image and the second training image into the multi-scale denoising model to be trained for denoising training to obtain a training denoised image corresponding to the second training image; and adjusting the model parameters of the multi-scale denoising model to be trained according to the loss between the training denoised image and the annotated image until convergence to obtain a trained multi-scale denoising model.

[0052] The method for determining that the multi-scale denoising model to be trained is trained to convergence may be: when the number of training times of the multi-scale denoising model to be trained reaches a preset number of training times, determining that the model parameters of the multi-scale denoising model to be trained are converged. Specifically, the preset number of training times provided in this embodiment may be a value such as 50,000 times or 100,000 times, and the specific value may be set according to actual application requirements, and is not specifically limited here.

[0053] In some embodiments, see Figure 2 , Figure 2 is a schematic diagram of a network structure of a multi-scale denoising model provided by an embodiment of the present invention, such as Figure 2 As shown, the multi-scale denoising model provided in this embodiment may include multiple convolutional layers with different resolutions, which may be as follows Figure 2 As shown in , it includes a full-resolution convolution layer, a 1 / 4-resolution convolution layer, a 1 / 16-resolution convolution layer, and a 1 / 64-resolution convolution layer. Specifically, the process of inputting the first image and the aligned image of different sizes provided in this embodiment into the multi-scale denoising model for denoising processing can be: inputting the first image to be denoised into the full-resolution convolution layer, and inputting the denoised aligned image into the minimum resolution convolution layer of the multi-scale denoising model, so as to gradually downsample the first image to 1 / 64 of the full resolution, and perform image super-resolution reconstruction on the first image at the 1 / 64-resolution convolution layer according to the denoised information of the aligned image, so as to gradually upsample to the original full resolution, thereby outputting the target image after denoising the first image.

[0054] It should be noted that the minimum resolution of the multi-scale denoising model provided in this embodiment is not limited to 1 / 64 of the full resolution (i.e., the resolution of the second transformed image after the size transformation process), but can also be any resolution. In addition, the position where the aligned image is input to the multi-scale denoising model is not limited to the lowest resolution of the model, but can also be any resolution convolution layer in the multi-scale denoising model. For example, Figure 2 In the process, when the aligned image is 1 / 64 of the full resolution and is input to the 1 / 16 or 1 / 4 or full resolution convolution layer of the multi-scale denoising model, the aligned image can be first upsampled to the resolution of the corresponding convolution layer, and then downsampled and upsampled step by step together with the first image. In this way, the denoising information in the denoised aligned image can be more effectively used to denoise the first image to be denoised, thereby further improving the denoising effect of the first image, and while improving the consistency of the denoising effect of the previous and next frames, the ghosting problem caused by the motion of the previous and next frames can also be reduced.

[0055] As an optional embodiment, since this embodiment requires the first frame of video frame image that has been subjected to noise reduction processing in the video to be denoised to be used as the second image, and when the first frame of video frame image of the video to be denoised has not been subjected to noise reduction processing, and the first image is also selected as the first frame of video frame image of the video to be denoised, this embodiment can directly perform noise reduction processing on the first frame of video frame image of the video to be denoised, and then use the first frame of video frame image after noise reduction processing as the second image, and perform noise reduction processing using the video noise reduction method provided by the embodiment of the present invention, thereby achieving noise reduction processing on the first frame of video frame image (i.e., the first image). Specifically, before the step of obtaining the first image to be denoised and the second image after noise reduction in the video to be denoised provided in this embodiment, the video noise reduction method provided in this embodiment may also include: when the first image is the first frame of video frame image in the video to be denoised, performing noise reduction processing on the first image to obtain a second noise reduction image; and determining the second noise reduction image as the second image.

[0056] At this point, the video denoising method provided by the embodiment of the present invention can be used to complete the denoising process on any frame of the video to be denoised, thereby achieving denoising process on the entire video to be denoised and obtaining the target video after denoising the video to be denoised.

[0057] To better illustrate the video noise reduction method provided by the embodiment of the present invention, please refer to Figure 3 , Figure 3 FIG. 2 is another flow chart of the video noise reduction method provided by an embodiment of the present invention. Figure 3As shown, firstly, the first frame of the video frame image after the denoising process in the video to be denoised is used as the second image, and then the second image and the first image to be denoised are transformed to obtain the first transformed image and the second transformed image with the same image size. Then, the first transformed image is denoised to obtain the first denoised image. The first denoised image and the second transformed image are subjected to motion estimation to obtain the motion estimation of the second transformed image relative to the first denoised image, and the corresponding alignment parameters are generated according to the motion estimation, so as to align the first denoised image and the second transformed image by the alignment parameters to obtain the aligned image corresponding to the second transformed avatar. Afterwards, the aligned image after denoising and the first image to be denoised are simultaneously input into the multi-scale denoising model, and the first image is denoised to obtain the target image after the denoising process of the first image. Then, the target image is continued to be used as the second image, and the image of the next frame of the original first image is used as the first image, so as to continue to perform denoising on the image to be denoised in the video to be denoised, until the denoising process of the entire video to be denoised is completed.

[0058] In summary, an embodiment of the present invention provides a video denoising method, which includes obtaining a first image to be denoised and a second image after denoising in a video to be denoised, wherein the first image is an arbitrary frame image of a video in the video to be denoised, the second image is an image of a frame before the first image, the size of the second image is not greater than the size of the first image, aligning the first image and the second image to obtain an aligned image corresponding to the second image, performing multi-scale denoising according to the first image and the aligned image to obtain a target image after denoising the first image, and replacing the first image in the video to be denoised with the target image. By adopting the embodiment of the present invention, the ghosting problem caused by multi-frame video denoising can be alleviated and avoided, and the denoising effect on the video can be effectively improved.

[0059] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a video noise reduction device. The video noise reduction device can be implemented as an independent entity or integrated in an electronic device, such as a terminal. The terminal can include a mobile phone, a tablet computer, etc.

[0060] See also Figure 4 , Figure 4 is a structural schematic diagram of a video noise reduction device provided by an embodiment of the present invention, such as Figure 4 As shown, the video noise reduction device 400 provided by the embodiment of the present invention includes: an acquisition module 401, an alignment module 402, and a first noise reduction module 403;

[0061] The acquisition module 401 is used to acquire a first image to be denoised and a second image after denoising in the video to be denoised, wherein the first image is any frame image of the video to be denoised, and the second image is a frame image before the first image.

[0062] The alignment module 402 is used to align the first image and the second image to obtain an aligned image corresponding to the second image.

[0063] The first denoising module 403 is configured to perform multi-scale denoising processing on the first image and the aligned image to obtain a target image after denoising the first image, and replace the first image in the video to be denoised with the target image.

[0064] In some embodiments, the first denoising module 403 provided in this embodiment is further specifically used to input the first image and the aligned image into a trained multi-scale denoising model to obtain a target image after denoising the first image.

[0065] In some embodiments, see Figure 5 , Figure 5 is another structural schematic diagram of a video noise reduction device provided by an embodiment of the present invention, such as Figure 5 As shown, the video noise reduction device provided in this embodiment may further include: a transformation module 404, a second noise reduction module 405, a determination module 406 and a training module 407;

[0066] The transformation module 404 is used to transform the first image and the second image to obtain a first transformed image and a second transformed image of the same size.

[0067] Specifically, the alignment module 402 provided in this embodiment is further used to perform alignment processing on the first transformed image and the second transformed image to obtain an aligned image corresponding to the second transformed image.

[0068] The second noise reduction module 405 is used to perform noise reduction processing on the first image to obtain a second noise reduction image when the first image is the first frame image of the video to be noise reduced.

[0069] The determination module 406 is configured to determine the second denoised image as the second image.

[0070] The training module 407 is used to obtain a training data set and a multi-scale denoising model to be trained, wherein the training data set includes a first training image of the first frame in the training video that has been denoised, a second training image of any frame that has not been denoised, and an annotated image of the second training image that has been denoised, and the first training image includes training images of multiple different sizes; the first training image and the second training image are input into the multi-scale denoising model to be trained for denoising training to obtain a training denoised image corresponding to the second training image; according to the loss between the training denoised image and the annotated image, the model parameters of the multi-scale denoising model to be trained are adjusted until convergence to obtain a trained multi-scale denoising model.

[0071] In some embodiments, the transformation processing provided in this embodiment may include downsampling processing. Then the transformation module 404 provided in this embodiment is specifically used to downsample the first image and the second image of the first size to obtain the first transformed image and the second transformed image of the second size, and the second size is not larger than the first size.

[0072] In some embodiments, the alignment module 402 provided in this embodiment is further specifically used to perform denoising on the first transformed image to obtain a first denoised image; perform motion estimation on the first denoised image and the second transformed image to obtain a motion estimation of the second transformed image relative to the first denoised image; and perform alignment on the first denoised image and the second transformed image based on the motion estimation to obtain an aligned image corresponding to the second transformed image.

[0073] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0074] Also, see Figure 6 , Figure 6 1 is a schematic diagram of a structure of an electronic device provided by an embodiment of the present invention, and the electronic device may be a mobile terminal such as a smart phone, a tablet computer, or the like. Figure 6 As shown, the electronic device 600 includes a processor 601 and a memory 602. The processor 601 is electrically connected to the memory 602.

[0075] The processor 601 is the control center of the electronic device 600. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device 600 and processes data by running or loading applications stored in the memory 602 and calling data stored in the memory 602, thereby monitoring the electronic device 600 as a whole.

[0076] In this embodiment, the processor 601 in the electronic device 600 will load the instructions corresponding to the processes of one or more applications into the memory 602 according to the following steps, and the processor 601 will run the application stored in the memory 602, thereby implementing any step in the video noise reduction method provided in the above embodiment.

[0077] The electronic device 600 can implement the steps in any embodiment of the video noise reduction method provided in the embodiments of the present invention, and therefore can achieve the beneficial effects that can be achieved by any video noise reduction method provided in the embodiments of the present invention. For details, please refer to the previous embodiments and will not be repeated here.

[0078] See also Figure 7 , Figure 7 is another structural schematic diagram of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, Figure 7 The electronic device 700 is a block diagram of a specific structure of an electronic device provided by an embodiment of the present invention, and the electronic device can be used to implement the video noise reduction method provided in the above embodiment. The electronic device 700 can be a mobile terminal such as a smart phone or a laptop computer.

[0079] The RF circuit 710 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 710 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 710 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0080] The memory 720 may be used to store software programs and modules, such as program instructions / modules corresponding to the video noise reduction method in the above-mentioned embodiment. The processor 780 executes various functional applications and performs video noise reduction by running the software programs and modules stored in the memory 720.

[0081] The memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 720 may further include a memory remotely arranged relative to the processor 780, and these remote memories may be connected to the electronic device 700 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0082] The input unit 730 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 730 may include a touch-sensitive surface 731 and other input devices 732. The touch-sensitive surface 731, also known as a touch display screen or touch pad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 731), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 731 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 780, and can receive and execute commands sent by the processor 780. In addition, the touch-sensitive surface 731 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 731, the input unit 730 may further include other input devices 732. Specifically, the other input devices 732 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0083] The display unit 740 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 700, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 740 may include a display panel 741, and optionally, the display panel 741 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 731 may cover the display panel 741, and when the touch-sensitive surface 731 detects a touch operation on or near it, it is transmitted to the processor 780 to determine the type of the touch event, and then the processor 780 provides corresponding visual output on the display panel 741 according to the type of the touch event. Although in the figure, the touch-sensitive surface 731 and the display panel 741 are implemented as two independent components to implement input and output functions, in some embodiments, the touch-sensitive surface 731 and the display panel 741 can be integrated to implement input and output functions.

[0084] The electronic device 700 may also include at least one sensor 750, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 741 according to the brightness of the ambient light, and the proximity sensor may generate an interrupt when the flip cover is closed or closed. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 700, they will not be repeated here.

[0085] The audio circuit 760, the speaker 761, and the microphone 762 can provide an audio interface between the user and the electronic device 700. The audio circuit 760 can transmit the electrical signal converted from the received audio data to the speaker 761, which is converted into a sound signal for output; on the other hand, the microphone 762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 760 and converted into audio data, and then the audio data is output to the processor 780 for processing, and then sent to another terminal through the RF circuit 710, or the audio data is output to the memory 720 for further processing. The audio circuit 760 may also include an earplug jack to provide communication between an external headset and the electronic device 700.

[0086] The electronic device 700 can help the user receive requests, send information, etc. through the transmission module 770 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 770 is shown in the figure, it is understandable that it is not a necessary component of the electronic device 700 and can be omitted as needed without changing the essence of the invention.

[0087] The processor 780 is the control center of the electronic device 700. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 720, and calling data stored in the memory 720, it executes various functions of the electronic device 700 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 780 may include one or more processing cores; in some embodiments, the processor 780 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 780.

[0088] The electronic device 700 also includes a power supply 790 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 780 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 790 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0089] Although not shown, the electronic device 700 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors to implement any step of the video noise reduction method provided in the above embodiment.

[0090] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0091] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores multiple instructions, and the instructions can be executed by the processor to implement any step of the video noise reduction method provided in the above embodiments.

[0092] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0093] Since the instructions stored in the storage medium can execute the steps in any embodiment of the video noise reduction method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any video noise reduction method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0094] The above is a detailed introduction to a video noise reduction method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application. Moreover, for those of ordinary skill in the art, without departing from the principles of the present invention, several improvements and modifications can be made, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. A video noise reduction method, characterized in that: include: Acquire a first image to be denoised and a second image after denoising in the video to be denoised, wherein the first image is any frame image of the video to be denoised, the second image is a frame image before the first image, and the size of the second image is not greater than the size of the first image; Performing alignment processing on the first image and the second image to obtain an aligned image corresponding to the second image; Multi-scale noise reduction processing is performed according to the first image and the aligned image to obtain a target image after the noise reduction of the first image, and the target image replaces the first image in the video to be denoised.

2. The method according to claim 1, characterized in that Before the step of aligning the first image and the second image to obtain an aligned image corresponding to the second image, the method further includes: Performing transformation processing on the first image and the second image to obtain a first transformed image and a second transformed image of the same size; The aligning the first image and the second image to obtain an aligned image corresponding to the second image includes: The first transformed image and the second transformed image are aligned to obtain an aligned image corresponding to the second transformed image.

3. The method according to claim 2, characterized in that The transformation process includes downsampling process; The transforming process is performed on the first image and the second image to obtain a first transformed image and a second transformed image of the same size, comprising: Downsampling is performed on the first image and the second image of a first size to obtain a first transformed image and a second transformed image of a second size, wherein the second size is not greater than the first size.

4. The method according to claim 2, characterized in that Before the step of aligning the first transformed image and the second transformed image to obtain an aligned image corresponding to the second transformed image, the method further includes: Performing denoising processing on the first transformed image to obtain a first denoised image; Performing motion estimation processing according to the first denoised image and the second transformed image to obtain a motion estimation of the second transformed image relative to the first denoised image; The aligning the first transformed image and the second transformed image to obtain an aligned image corresponding to the second transformed image includes: According to the motion estimation, the first denoised image and the second transformed image are aligned to obtain an aligned image corresponding to the second transformed image.

5. The method according to claim 1, characterized in that Before the step of obtaining the first image to be denoised and the second image after denoising in the video to be denoised, the method further includes: When the first image is a first frame image of a video to be denoised, performing denoising processing on the first image to obtain a second denoised image; The second denoised image is determined as a second image.

6. The method according to claim 1, characterized in that The performing multi-scale noise reduction processing according to the first image and the aligned image to obtain a target image after the noise of the first image is reduced includes: Inputting the first image and the aligned image into a trained multi-scale denoising model to obtain a target image after denoising the first image; Before the step of performing multi-scale noise reduction processing on the first image and the aligned image to obtain a target image after noise reduction of the first image, the method further includes: Obtaining a training data set and a multi-scale denoising model to be trained, wherein the training data set includes a first training image of a first frame in a training video that has been denoised, a second training image of any frame that has not been denoised, and an annotated image of the second training image that has been denoised, and the first training image includes training images of multiple different sizes; Inputting the first training image and the second training image into the multi-scale denoising model to be trained to perform denoising training, so as to obtain a training denoised image corresponding to the second training image; According to the loss between the training denoised image and the annotated image, the model parameters of the multi-scale denoising model to be trained are adjusted until convergence, so as to obtain a trained multi-scale denoising model.

7. A video noise reduction device, characterized in that: include: An acquisition module, used to acquire a first image to be denoised and a second image after denoising in the video to be denoised, wherein the first image is any frame image of the video to be denoised, and the second image is a frame image previous to the first image; an alignment module, configured to perform alignment processing on the first image and the second image to obtain an aligned image corresponding to the second image; The first denoising module is used to perform multi-scale denoising processing on the first image and the aligned image to obtain a target image after denoising the first image, and replace the first image in the video to be denoised with the target image.

8. The device according to claim 7, characterized in that The video noise reduction device also includes: A transformation module, used for performing transformation processing on the first image and the second image to obtain a first transformed image and a second transformed image of the same size; The alignment module is further used to perform alignment processing on the first transformed image and the second transformed image to obtain an aligned image corresponding to the second transformed image.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 6 are implemented.

Images