Image noise reduction method and device, electronic equipment and storage medium

Abstract

The invention discloses an image denoising method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-denoised image set which comprises a plurality of frames of noise-containing images obtained by photographing the same object in the same scene, and carrying out the denoising of a first image and a second image in the to-be-denoised image set according to the first image and the second image in the to-be-denoised image set. A feature extractor in a preset convolutional neural network is adjusted, an adjusted target feature extractor is obtained, the target feature extractor is used for performing corresponding convolutional feature extraction processing on the first image and the second image, the target feature extractor is called to perform convolutional feature extraction processing on the first image and the second image, and the first image and the second image are obtained; and carrying out noise reduction processing on the target feature image to obtain a target image after the first image is subjected to noise reduction. By adopting the embodiment of the invention, the problem that the noise reduction effect is influenced by alignment processing can be avoided, so that the noise reduction effect is improved when noise reduction processing is carried out on multiple frames of images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image noise reduction method, device, electronic equipment and storage medium. Background Art

[0002] Image noise is a random variation in brightness or color information in an image (not present in the photographed object itself), usually a manifestation of electronic noise. It is generally generated by the sensors and circuits of scanners or digital cameras, or by film grain or the inevitable shot noise in ideal photodetectors. Image noise is an undesirable byproduct of the image capture process, which brings errors and additional information to the image. Therefore, eliminating image noise by performing noise reduction processing on the image is a key step to improve image quality.

[0003] In the related art, when dealing with noise reduction problems involving multiple frames of images, it is necessary to consider the deviation caused by the motion of other frames relative to the reference frame, so it is necessary to introduce a motion estimation module. There are two ways to process the obtained motion estimation. One way is to align the images of other frames relative to the reference frame based on the motion estimation, and then perform noise reduction on the aligned multi-frame images. Most multi-frame-based technologies choose this method, but the alignment operation will bring a series of problems. First, if the original image is in the RAW domain, there may be pattern errors after alignment, which is harmful to multi-frame noise reduction; secondly, since the alignment method cannot be 100% accurate, the unaligned parts will interfere with the multi-frame noise reduction, thereby affecting the multi-frame noise reduction effect. Another way to process is to sample according to the alignment results when the information of the corresponding objects on other frames is needed. There is also a problem that the motion estimation cannot be 100% accurate, which will interfere with the motion estimation failure area and affect the noise reduction effect. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide an image noise reduction method, device, electronic device and storage medium to solve the technical problem that the noise reduction effect cannot be effectively improved when performing noise reduction processing on multiple frames of images.

[0005] In a first aspect, an embodiment of the present invention provides an image noise reduction method, comprising:

[0006] Acquire a set of images to be denoised, wherein the set of images to be denoised includes multiple frames of noisy images obtained by photographing the same object in the same scene;

[0007] According to the first image and the second image in the image set to be denoised, a feature extractor in a preset convolutional neural network is adjusted to obtain an adjusted target feature extractor, wherein the first image and the second image are different images, and the target feature extractor is used to perform corresponding convolution feature extraction processing on the first image and the second image;

[0008] Calling the target feature extractor to perform convolution feature extraction processing on the first image and the second image to obtain an extracted target feature image;

[0009] The target feature image is subjected to denoising processing to obtain a target image after denoising the first image.

[0010] In a second aspect, an embodiment of the present invention provides an image noise reduction device, comprising:

[0011] An acquisition module is used to acquire a set of images to be denoised, wherein the set of images to be denoised includes multiple frames of noisy images obtained by photographing the same object in the same scene;

[0012] An adjustment module, configured to adjust a feature extractor in a preset convolutional neural network according to a first image and a second image in the image set to be denoised, to obtain an adjusted target feature extractor, wherein the first image and the second image are different images, and the target feature extractor is configured to perform corresponding convolution feature extraction processing on the first image and the second image;

[0013] A feature extraction module, used for calling the target feature extractor to perform convolution feature extraction processing on the first image and the second image to obtain an extracted target feature image;

[0014] The denoising module is used to perform denoising on the target feature image to obtain a target image after denoising the first image.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned image denoising methods when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any one of the above-mentioned image denoising methods are implemented.

[0017] The embodiments of the present invention provide an image denoising method, device, electronic device and storage medium. The method adjusts a feature extractor through a noisy first image and a second image so that the adjusted target feature extractor can perform corresponding convolution feature extraction processing on different images to output feature images containing more effective image feature information; and avoids the use of alignment processing, thereby avoiding the problem of affecting the denoising effect caused by the alignment processing, so that the target feature image provided by this embodiment is used for denoising processing, which can effectively improve the denoising effect of the first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of an image noise reduction method provided by an embodiment of the present invention;

[0019] Figure 2 It is a schematic diagram of a flow chart of adjusting a feature extractor provided by an embodiment of the present invention;

[0020] Figure 3 It is a schematic diagram of a flow chart of adjusting a convolution kernel in a feature extractor provided by an embodiment of the present invention;

[0021] Figure 4 It is a schematic diagram of a flow chart of convolution feature extraction processing provided by an embodiment of the present invention;

[0022] Figure 5 is a structural schematic diagram of an image noise reduction device provided by an embodiment of the present invention;

[0023] Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present invention;

[0024] Figure 7 This is another structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0028] In the related art, when dealing with noise reduction problems involving multiple frames of images, it is necessary to consider the deviation caused by the motion of other frames relative to the reference frame, so it is necessary to introduce a motion estimation module. There are two ways to process the obtained motion estimation. One way is to align the images of other frames relative to the reference frame based on the motion estimation, and then perform noise reduction on the aligned multi-frame images. Most multi-frame-based technologies choose this method, but the alignment operation will bring a series of problems. First, if the original image is in the RAW domain, there may be pattern errors after alignment, which is harmful to multi-frame noise reduction; secondly, since the alignment method cannot be 100% accurate, the unaligned parts will interfere with the multi-frame noise reduction, thereby affecting the multi-frame noise reduction effect. Another way to process is to sample according to the alignment results when the information of the corresponding objects on other frames is needed. There is also a problem that the motion estimation cannot be 100% accurate, which will interfere with the motion estimation failure area and affect the noise reduction effect.

[0029] In order to solve the technical problems existing in the related art, an embodiment of the present invention provides an image noise reduction method, which can be applied to electronic devices. When an object is photographed by a camera of an electronic device, due to limitations such as hardware and environment, the acquired video may have noise problems such as noise points, patches, ghosting, etc. that affect the visual effect. At this time, the denoising process can be performed on the undenoised image collected by the camera in the electronic device to obtain a denoised image.

[0030] Among them, the above-mentioned electronic device can be mobile or fixed. For example, the electronic device can be a mobile phone with image noise reduction function, a camera, a camcorder, a vehicle, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a smart watch, augmented reality (AR) / virtual reality (VR), a wearable device (WD), a game console, etc. The embodiment of the present application does not limit the specific type of the electronic device.

[0031] For details, see Figure 1 , Figure 1 1 is a flow chart of an image noise reduction method provided by an embodiment of the present invention, the method comprising steps 101 to 104;

[0032] Step 101: Obtain a set of images to be denoised.

[0033] Among them, the image set to be denoised provided in this embodiment includes multiple frames of noisy images obtained by shooting the same object in the same scene. The noisy image may include but is not limited to the original RAW image of the sensor in the image signal processing (ImageSignal Process, ISP) process, and the color image after ISP processing, etc. As long as the image has multiple frame noise reduction requirements, the image noise reduction method provided by the embodiment of the present invention can be used to improve the effect of multi-frame noise reduction; in addition, the image noise reduction method provided by the embodiment of the present invention is also applicable to video stream processing. Specifically, when the image set to be denoised contains two frames of images, one frame is a noisy image of the current frame, and the other frame can be either a noise-reduced image of the previous frame of the current frame, or a noisy image of any frame. When used for video stream processing, the image noise reduction method provided by the embodiment of the present invention can not only improve the noise reduction effect through multi-frame noise reduction, but also because of the addition of the previous frame, it is conducive to maintaining the consistency between frames, and it is not easy for noise to jump between frames, which further improves the noise reduction effect.

[0034] It should be noted that the number of images included in the image set to be denoised provided in this embodiment is multiple, including but not limited to 2, 5, 7 and other values. As long as the number is more than 2, that is, N frames of noisy images, N≥2.

[0035] Step 102: According to the first image and the second image in the image set to be denoised, a feature extractor in a preset convolutional neural network is adjusted to obtain an adjusted target feature extractor.

[0036] Among them, the first image and the second image provided in this embodiment are different images, and the target feature extractor is mainly used to perform corresponding convolution feature extraction processing on the first image and the second image.

[0037] In this embodiment, the first image can be a reference frame image, that is, an image that needs to be denoised, and the second image can be an image in the image set to be denoised except the first image, that is, a noisy supplementary frame image. The second image is mainly used to assist the first image in performing multi-frame denoising processing to achieve the denoising purpose of the first image.

[0038] In some embodiments, since this embodiment mainly performs noise reduction processing on the first image, please refer to Figure 2, Figure 2 is a schematic diagram of a flow chart of adjusting a feature extractor provided by an embodiment of the present invention, such as Figure 2 As shown, the step of adjusting the feature extractor, that is, the step of adjusting the feature extractor in the preset convolutional neural network according to the first image and the second image in the image set to be denoised, to obtain the adjusted target feature extractor, may include steps 201 to 203;

[0039] Step 201: Determine a first image and a second image in a set of images to be denoised.

[0040] In some embodiments, in order to improve the noise reduction effect of multi-frame noise reduction, this embodiment needs to first determine the reference frame image, that is, determine the first image that needs to be processed for noise reduction. Specifically, the steps of determining the first image and the second image in the image set to be denoised provided by this embodiment can be: sorting each frame of the image set to be denoised according to the exposure time of the shooting; determining the image in the middle of the sorting as the first image; and determining the images in the image set to be denoised except the first image as the second image.

[0041] Among them, the exposure time provided by this embodiment for photographing the same object can be determined according to the exposure time during the photographing, and specifically can include long exposure time, normal exposure time and short exposure time. For example, by photographing the same object with different exposure times, J frames of short exposure time images, K frames of normal exposure time images and L frames of long exposure time images can be obtained, where J, K, and L can all be values ​​greater than or equal to 0. Optionally, this embodiment adopts J=K=2, L=1. Then this embodiment can first sort all images according to the length of exposure time, and then select the image in the middle of the sorting as the reference frame image, that is, the first image. Specifically, this embodiment selects one frame of the K frames of normal exposure time images as the reference frame image, and the remaining frames as supplementary frame images, that is, the second image.

[0042] In one implementation, there are two images in the middle of the ranking, that is, J=K=L=2. In this case, the step of determining the image in the middle of the ranking as the first image provided by this embodiment can be: when there are two images in the middle of the ranking, calculate the gradient of the green channel in the two images in the middle of the ranking; determine the image with the larger green gradient as the first image. That is, calculate the gradient of the green channel in K frames of normal exposure time images, and then select the frame with the largest gradient as the reference frame image, that is, the first image.

[0043] Step 202, determining a motion estimate of the second image relative to the first image.

[0044] In this embodiment, the motion estimation provided by this embodiment can be an offset feature map of each second image compared to the first image, and the offset feature map can include the direction of the offset and the size of the offset. In one embodiment, the feature points of each image can be extracted by a feature extraction algorithm, and then the motion formula of each second image compared to the first image can be calculated by a feature point matching method, thereby calculating the offset feature map of each pixel on each second image compared to the same pixel on the first image. In another embodiment, the offset feature map of each second image compared to the first image is directly calculated by a traditional or neural network-based optical flow model.

[0045] Step 203, adjusting the feature extractor in the preset convolutional neural network according to the motion estimation to obtain an adjusted target feature extractor.

[0046] Among them, the feature extractor in the preset convolutional neural network provided in this embodiment can be a feature extractor in any convolutional neural network that can perform convolutional feature extraction processing on an image, such as feature extractors in networks such as SPP net, Fast RCNN, and Faster RCNN. Therefore, the feature extractor of this embodiment is not specifically limited here.

[0047] In this embodiment, by determining the motion estimate of the second image relative to the first image, the feature extractor can be adjusted according to the determined motion estimate to obtain a target feature extractor, so that the adjusted target feature extractor can perform corresponding convolution feature extraction processing on the second image based on the determined motion estimate. Specifically, the target feature extractor provided in this embodiment can mainly change the position and / or size of the convolution kernel according to different motion estimates to achieve accurate convolution feature extraction processing for target objects at different positions in different second images. Among them, the target object is an object that is photographed in the same scene and displayed in each frame of the image set to be denoised, that is, the target object is the same object photographed.

[0048] In some embodiments, when denoising multiple frames, there may be displacement between the frames that affects the denoising effect. Figure 3 , Figure 3 It is a flowchart of adjusting the convolution kernel in the feature extractor provided by an embodiment of the present invention, that is, the step of adjusting the feature extractor in the preset convolutional neural network according to motion estimation provided by this embodiment to obtain the adjusted target feature extractor may include steps 301 to 302;

[0049] Step 301, determining a set of convolution kernel sizes according to the size of the motion estimation of each second image relative to the first image.

[0050] Among them, the size of the motion estimation provided by this embodiment is proportional to the convolution kernel size. That is, when the motion estimation corresponding to the second image is larger, the size of the convolution kernel used when performing convolution feature extraction processing on the second image is also larger. The convolution kernel size set provided by this embodiment includes the convolution kernel size corresponding to each second image, so that each second image can be processed by the corresponding convolution kernel size when performing convolution feature extraction processing, effectively ensuring accurate convolution feature extraction processing of the position of the target object in each second image.

[0051] Step 302, according to the convolution kernel size set, adjust the convolution kernel of the feature extractor in the preset convolutional neural network to obtain an adjusted target feature extractor.

[0052] This embodiment adjusts the convolution kernel of the feature extractor in the preset convolutional neural network so that the adjusted target feature extractor can use the corresponding convolution kernel when performing convolution feature extraction processing on different second images, thereby achieving accurate convolution feature extraction processing on the same target object in multiple frames of images, so as to improve the noise reduction effect of subsequent noise reduction processing on the first image.

[0053] Specifically, this embodiment changes the convolution kernel size of the feature extractor by the size of the motion estimate, so as to guide the feature extractor to extract the features of the target object in different second images. In this way, there is no need to perform an alignment step before performing multi-frame image denoising, as in the multi-frame image denoising method in the related art, thereby effectively preventing the occurrence of erroneous information due to insufficient alignment accuracy during the alignment process and damaging the denoising effect; in addition, this embodiment also takes into account the situation where there is a large motion between the multi-frame images, and for the second image with a large motion estimate, the convolution kernel size is correspondingly increased when performing convolution feature extraction processing on it, thereby effectively improving the overall robustness of the denoising to large motion.

[0054] Step 103, calling a target feature extractor to perform convolution feature extraction processing on the first image and the second image to obtain an extracted target feature image.

[0055] In this embodiment, since this embodiment has determined different convolution kernel sizes for motion estimation of different second images and configured all convolution kernel sizes into the target feature extractor, the target feature extractor can be directly called to perform convolution feature extraction processing on different images using corresponding convolution kernels to extract more feature information about the target object in the image.

[0056] In some embodiments, in order to obtain more feature information about the target object in the image, before the step of calling the target feature extractor provided in the present embodiment to perform convolution feature extraction processing on the first image and the second image to obtain the extracted target feature image, the image denoising method provided in the present embodiment may also include: determining the convolution kernel with the smallest size in the convolution kernel size set as the first convolution kernel used when performing convolution feature extraction processing on the first image; determining the position and size of the second convolution kernel used when performing convolution feature extraction processing on each second image according to the direction and size of the motion estimation of each second image relative to the first image.

[0057] By using the convolution kernel of the smallest size in the convolution kernel size set determined in the above embodiment as the first convolution kernel used when performing convolution feature extraction processing on the first image, the convolution efficiency of the convolution feature extraction processing on the first image can be effectively improved. Specifically, during the actual convolution feature extraction processing, the convolution feature extraction processing on the first image can be performed by stacking multiple identical first convolution kernels to provide a larger receptive field, so that more image feature information can be extracted.

[0058] According to the direction of motion estimation of each second image relative to the first image, the position of the second convolution kernel can be determined, that is, the position of the convolution feature extraction process for different second images can be determined, so as to accurately perform convolution processing on the target object. According to the size of the motion estimation, the size of the convolution kernel used when performing convolution feature extraction processing on the second image is changed accordingly, which can provide better robustness for the noise reduction processing when there is large motion between multiple frames of images, thereby improving the noise reduction effect.

[0059] After determining the corresponding convolution kernel for different images, see Figure 4 , Figure 4 is a flow chart of the convolution feature extraction process provided by an embodiment of the present invention, that is, the step of calling the target feature extractor to perform convolution feature extraction on the first image and the second image to obtain the extracted target feature image provided by the present embodiment, which may include steps 401 to 403;

[0060] Step 401, calling a target feature extractor to adopt a first convolution kernel to perform convolution feature extraction processing on a first image to obtain an extracted first feature image.

[0061] Step 402, calling the target feature extractor to adopt the second convolution kernel to perform convolution feature extraction processing on the corresponding second image to obtain an extracted second feature image.

[0062] Step 403: Combine the first feature image and the second feature image to obtain a target feature image.

[0063] As an optional embodiment, the convolution kernel size set provided by this embodiment may contain multiple convolution kernel sizes. In this case, this embodiment may set a corresponding selection threshold according to the number of different convolution kernel sizes in the convolution kernel size set, so as to adaptively select a suitable convolution kernel size through different motion estimates and the set selection threshold. If there is only one convolution kernel size in the convolution kernel size set, it is not necessary to set a selection threshold; if there are two different convolution kernel sizes in the convolution kernel size set, such as a large and a small convolution kernel size, then a selection threshold needs to be set at this time, so as to select a larger convolution kernel size when the size of the motion estimate exceeds the selection threshold, and select a smaller convolution kernel size when the size of the motion estimate does not exceed the selection threshold; if there are X different convolution kernel sizes in the convolution kernel size set, it is necessary to set X-1 selection thresholds that increase in sequence, so as to adaptively select an accurate convolution kernel size for convolution operation according to the size of different motion estimates.

[0064] Among them, in this embodiment, the offset between the same pixel in the second image and the first image can be used as the size of the motion estimation, so as to select the corresponding convolution kernel size according to the size of the pixel. For example, according to the size of the motion estimation of each second image relative to the first image, the convolution kernel size set K and the corresponding selection threshold set T used for convolution feature extraction processing are determined. In this embodiment, the largest convolution kernel size in K is first determined. This size must meet the information of the maximum distance position in the motion estimation. After M times (M is greater than or equal to 1) of convolution with the largest size convolution kernel, it is aggregated into the information of the same pixel point in the first image. When M=2, if the size of the motion estimation is within 2 pixels, then in the process of convolution feature extraction processing, there is no need to change the size of the convolution kernel. At this time, K={3*3}, and T is an empty set. If the size of the motion estimation exceeds 2 pixels but is less than 5 pixels, then K = {3*3, 5*5}, T = {2}. At this time, when performing convolution feature extraction processing on different second images, if the motion vector of the pixel point on the first image on the second image is less than or equal to 2 pixels, the convolution kernel size corresponding to the second image is 3*3. If it is greater than 2 pixels, the convolution kernel size corresponding to the second image is 5*5. If the size of the motion estimation of the second image is less than 2 pixels, more than 2 pixels but less than 5 pixels, or more than 5 pixels but less than 7 pixels, then K = {3*3, 5*5, 7*7}, T = {2, 5}. At this time, when performing convolution feature extraction processing on different second images, if the motion vector of the pixel on the first image on the second image is less than or equal to 2 pixels, the convolution kernel size corresponding to the second image is 3*3, if it is greater than 2 pixels but less than 5 pixels, the convolution kernel size corresponding to the second image is 5*5, and if it is greater than 5 pixels but less than 7 pixels, the convolution kernel size corresponding to the second image is 7*7. By analogy, different numbers of selection thresholds can be set according to different motion estimation sizes, so that the target feature extractor can adaptively select the corresponding selection threshold for a variety of different convolution kernel sizes to achieve accurate convolution operations.

[0065] It should be noted that, in order to ensure that the largest convolution kernel in the convolution kernel size set can perform normal convolution operations on the image, before the step of calling the target feature extractor provided in this embodiment to use the first convolution kernel to perform convolution feature extraction processing on the first image to obtain the extracted first feature image, the image denoising method provided in this embodiment may also include: filling the pixels at the edge positions of the first image and the second image according to the convolution kernel size set with the largest size. For example, when the determined convolution kernel size set K = {3*3, 5*5}, and the threshold set T = {2} is selected, at this time, each image (including the first image and the second image) is padded with a size of 2 around it, and the padding value is 0, so as to ensure that the largest convolution kernel in the convolution kernel size set can perform normal convolution operations on the image.

[0066] After completing the method steps of the above embodiment, the first image and the second image can be subjected to convolution feature extraction processing to obtain an extracted target feature image. Specifically, in the process of convolution feature extraction processing, each position on the first image is subjected to convolution operation according to the minimum value in the convolution kernel size set K, that is, 3*3 is the convolution kernel size; for the second image, it is necessary to first determine the position of each pixel point on the first image on the second image according to motion estimation. At this time, if the position on the second image determined by motion estimation exceeds the edge position before the filling process, it is corrected to the edge position in the same direction before the filling according to the motion estimation, and the distance between the position on the second image after correction and the corresponding pixel point on the first image is compared with the threshold in the threshold set T, thereby determining the convolution kernel size of the convolution kernel used, thereby effectively ensuring the normal convolution operation of the target feature extractor.

[0067] Step 104 , performing noise reduction processing on the target feature image to obtain a target image after noise reduction of the first image.

[0068] In this embodiment, this embodiment can use a spatiotemporal convolutional neural network to perform denoising based on deep learning, so as to output a target image after denoising the first image. A traditional multi-frame denoising algorithm or other denoising models can also be used for denoising, which is not specifically limited here.

[0069] As an optional embodiment, when a denoising model is used to perform denoising on a target feature image, this embodiment can combine a preset convolutional neural network and the adopted denoising model into a multi-frame denoising model. In this way, the long process of separate training can be omitted during the training of the model, thereby improving the training efficiency of the model.

[0070] In summary, an embodiment of the present invention provides an image denoising method, which includes obtaining a set of images to be denoised, the set of images to be denoised includes multiple frames of noisy images obtained by photographing the same object in the same scene, adjusting a feature extractor in a preset convolutional neural network according to a first image and a second image in the set of images to be denoised, obtaining an adjusted target feature extractor, the target feature extractor is used to perform corresponding convolution feature extraction processing on the first image and the second image, calling the target feature extractor to perform convolution feature extraction processing on the first image and the second image to obtain an extracted target feature image, performing denoising processing on the target feature image, and obtaining a target image after denoising the first image. By adopting the embodiment of the present invention, the problem of affecting the denoising effect caused by the alignment processing can be avoided, thereby improving the denoising effect when denoising multiple frames of images.

[0071] According to the method described in the above embodiment, this embodiment will be further described from the perspective of an image noise reduction device. The image noise reduction device can be implemented as an independent entity or integrated in an electronic device, such as a terminal. The terminal may include a mobile phone, a tablet computer, etc.

[0072] See also Figure 5 , Figure 5 is a structural schematic diagram of an image noise reduction device provided by an embodiment of the present invention, such as Figure 5 As shown, the image denoising device 500 provided by the embodiment of the present invention includes: an acquisition module 501, an adjustment module 502, a feature extraction module 503, and a denoising module 504;

[0073] The acquisition module 501 is used to acquire a set of images to be denoised.

[0074] The image set to be denoised includes multiple frames of noisy images obtained by photographing the same object in the same scene.

[0075] The adjustment module 502 is used to adjust the feature extractor in the preset convolutional neural network according to the first image and the second image in the image set to be denoised, so as to obtain an adjusted target feature extractor.

[0076] The first image and the second image are different images, and the target feature extractor is used to perform corresponding convolution feature extraction processing on the first image and the second image.

[0077] In some embodiments, the adjustment module 502 provided in this embodiment is specifically used to: determine the first image and the second image in the image set to be denoised; determine the motion estimation of the second image relative to the first image; and adjust the feature extractor in the preset convolutional neural network according to the motion estimation to obtain an adjusted target feature extractor.

[0078] In some embodiments, the adjustment module 502 provided in this embodiment is further specifically used to: sort each frame of the image in the set of images to be denoised according to the exposure time of the shooting; determine the image in the middle of the sorting as the first image; and determine the images in the set of images to be denoised except the first image as the second image.

[0079] In some embodiments, the adjustment module 502 provided in this embodiment is further specifically used to: when there are two images in the middle of the sort, calculate the gradient of the green channel in the two images in the middle of the sort; and determine the image with the larger green gradient as the first image.

[0080] In some embodiments, the adjustment module 502 provided in this embodiment is further specifically used to: determine a convolution kernel size set based on the size of the motion estimate of each second image relative to the first image, the size of the motion estimate being proportional to the convolution kernel size; and adjust the convolution kernel of the feature extractor in a preset convolutional neural network based on the convolution kernel size set to obtain an adjusted target feature extractor.

[0081] In some embodiments, the image noise reduction device 500 provided in this embodiment further includes a first determination module and a second determination module;

[0082] Among them, the first determination module is used to determine the convolution kernel with the smallest size in the convolution kernel size set as the first convolution kernel used when performing convolution feature extraction processing on the first image.

[0083] The second determination module is used to determine the position and size of the second convolution kernel used when performing convolution feature extraction processing on each second image according to the direction and size of the motion estimation of each second image relative to the first image.

[0084] The feature extraction module 503 is used to call the target feature extractor to perform convolution feature extraction processing on the first image and the second image to obtain an extracted target feature image.

[0085] In this embodiment, the feature extraction module 503 provided in this embodiment is specifically used to: call the target feature extractor to adopt the first convolution kernel, perform convolution feature extraction processing on the first image, and obtain an extracted first feature image; call the target feature extractor to adopt the second convolution kernel, perform convolution feature extraction processing on the corresponding second image, and obtain an extracted second feature image; combine the first feature image and the second feature image to obtain a target feature image.

[0086] In this embodiment, the feature extraction module 503 provided in this embodiment is further specifically used to: fill the pixels at the edge positions of the first image and the second image according to the convolution kernel of the largest size in the convolution kernel size set.

[0087] The denoising module 504 is used to perform denoising processing on the target feature image to obtain a target image after denoising the first image.

[0088] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0089] Also, see Figure 6 , Figure 6 1 is a schematic diagram of a structure of an electronic device provided by an embodiment of the present invention, and the electronic device may be a mobile terminal such as a smart phone, a tablet computer, or the like. Figure 6 As shown, the electronic device 600 includes a processor 601 and a memory 602. The processor 601 is electrically connected to the memory 602.

[0090] The processor 601 is the control center of the electronic device 600. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device 600 and processes data by running or loading applications stored in the memory 602 and calling data stored in the memory 602, thereby monitoring the electronic device 600 as a whole.

[0091] In this embodiment, the processor 601 in the electronic device 600 will load the instructions corresponding to the processes of one or more applications into the memory 602 according to the following steps, and the processor 601 will run the application stored in the memory 602, thereby implementing any step in the image denoising method provided in the above embodiment.

[0092] The electronic device 600 can implement the steps in any embodiment of the image noise reduction method provided in the embodiments of the present invention, and therefore can achieve the beneficial effects that can be achieved by any image noise reduction method provided in the embodiments of the present invention. For details, please refer to the previous embodiments and will not be repeated here.

[0093] See also Figure 7 , Figure 7 is another structural schematic diagram of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, Figure 7 The specific structural block diagram of the electronic device provided by the embodiment of the present invention is shown, and the electronic device can be used to implement the image noise reduction method provided in the above embodiment. The electronic device 700 can be a mobile terminal such as a smart phone or a laptop computer.

[0094] The RF circuit 710 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 710 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 710 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0095] The memory 720 may be used to store software programs and modules, such as program instructions / modules corresponding to the image noise reduction method in the above-mentioned embodiment. The processor 780 executes various functional applications and performs image noise reduction by running the software programs and modules stored in the memory 720.

[0096] The memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 720 may further include a memory remotely arranged relative to the processor 780, and these remote memories may be connected to the electronic device 700 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0097] The input unit 730 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 730 may include a touch-sensitive surface 731 and other input devices 732. The touch-sensitive surface 731, also known as a touch display screen or touch pad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 731), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 731 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 780, and can receive and execute commands sent by the processor 780. In addition, the touch-sensitive surface 731 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 731, the input unit 730 may further include other input devices 732. Specifically, the other input devices 732 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0098] The display unit 740 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 700, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 740 may include a display panel 741, and optionally, the display panel 741 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 731 may cover the display panel 741, and when the touch-sensitive surface 731 detects a touch operation on or near it, it is transmitted to the processor 780 to determine the type of the touch event, and then the processor 780 provides corresponding visual output on the display panel 741 according to the type of the touch event. Although in the figure, the touch-sensitive surface 731 and the display panel 741 are implemented as two independent components to implement input and output functions, in some embodiments, the touch-sensitive surface 731 and the display panel 741 can be integrated to implement input and output functions.

[0099] The electronic device 700 may also include at least one sensor 750, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 741 according to the brightness of the ambient light, and the proximity sensor may generate an interrupt when the flip cover is closed or closed. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 700, they will not be repeated here.

[0100] The audio circuit 760, the speaker 761, and the microphone 762 can provide an audio interface between the user and the electronic device 700. The audio circuit 760 can transmit the electrical signal converted from the received audio data to the speaker 761, which is converted into a sound signal for output; on the other hand, the microphone 762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 760 and converted into audio data, and then the audio data is output to the processor 780 for processing, and then sent to another terminal through the RF circuit 710, or the audio data is output to the memory 720 for further processing. The audio circuit 760 may also include an earplug jack to provide communication between an external headset and the electronic device 700.

[0101] The electronic device 700 can help the user receive requests, send information, etc. through the transmission module 770 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 770 is shown in the figure, it is understandable that it is not a necessary component of the electronic device 700 and can be omitted as needed without changing the essence of the invention.

[0102] The processor 780 is the control center of the electronic device 700. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 720, and calling data stored in the memory 720, it executes various functions of the electronic device 700 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 780 may include one or more processing cores; in some embodiments, the processor 780 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 780.

[0103] The electronic device 700 also includes a power supply 790 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 780 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 790 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0104] Although not shown, the electronic device 700 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors to implement any step of the image noise reduction method provided in the above embodiment.

[0105] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0106] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores multiple instructions, and the instructions can be executed by the processor to implement any step of the image denoising method provided in the above embodiments.

[0107] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0108] Since the instructions stored in the storage medium can execute the steps in any embodiment of the image denoising method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any image denoising method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0109] The above is a detailed introduction to an image noise reduction method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application. Moreover, for those of ordinary skill in the art, without departing from the principles of the present invention, several improvements and modifications can be made, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. An image denoising method, characterized in that: include: Acquire a set of images to be denoised, wherein the set of images to be denoised includes multiple frames of noisy images obtained by photographing the same object in the same scene; According to the first image and the second image in the image set to be denoised, a feature extractor in a preset convolutional neural network is adjusted to obtain an adjusted target feature extractor, wherein the first image and the second image are different images, and the target feature extractor is used to perform corresponding convolution feature extraction processing on the first image and the second image; Calling the target feature extractor to perform convolution feature extraction processing on the first image and the second image to obtain an extracted target feature image; The target feature image is subjected to denoising processing to obtain a target image after denoising the first image.

2. The method according to claim 1, characterized in that: The method of adjusting a feature extractor in a preset convolutional neural network according to the first image and the second image in the image set to be denoised to obtain an adjusted target feature extractor comprises: Determine a first image and a second image in the set of images to be denoised; determining a motion estimate of the second image relative to the first image; The feature extractor in a preset convolutional neural network is adjusted according to the motion estimation to obtain an adjusted target feature extractor.

3. The method according to claim 2, characterized in that The determining of the first image and the second image in the set of images to be denoised comprises: Sorting each frame of the image in the set of images to be denoised according to the exposure time of the shooting; The image in the middle of the ranking is determined as the first image; The images in the set of images to be denoised except the first image are determined as second images.

4. The method according to claim 3, characterized in that: The step of determining the image in the middle of the ranking as the first image comprises: When there are two images in the middle of the ranking, the gradients of the green channels of the two images in the middle of the ranking are calculated; The image with a larger green gradient is determined as the first image.

5. The method according to claim 2, characterized in that: The step of adjusting a feature extractor in a preset convolutional neural network according to the motion estimation to obtain an adjusted target feature extractor includes: Determine a convolution kernel size set according to a size of motion estimation of each second image relative to the first image, wherein the size of the motion estimation is proportional to the convolution kernel size; According to the convolution kernel size set, the convolution kernel of the feature extractor in the preset convolutional neural network is adjusted to obtain an adjusted target feature extractor.

6. The method according to claim 5, characterized in that Before the step of calling the target feature extractor to perform convolution feature extraction processing on the first image and the second image to obtain an extracted target feature image, the method further includes: Determining the convolution kernel with the smallest size in the convolution kernel size set as the first convolution kernel used when performing convolution feature extraction processing on the first image; Determining, according to the estimated direction and size of the motion of each of the second images relative to the first image, the position and size of the second convolution kernel used when performing convolution feature extraction processing on each of the second images; The calling of the target feature extractor to perform convolution feature extraction processing on the first image and the second image to obtain an extracted target feature image includes: Calling the target feature extractor to use the first convolution kernel to perform convolution feature extraction processing on the first image to obtain an extracted first feature image; Calling the target feature extractor to use the second convolution kernel to perform convolution feature extraction processing on the corresponding second image to obtain an extracted second feature image; The first feature image and the second feature image are combined to obtain a target feature image.

7. The method according to claim 6, characterized in that Before the step of calling the target feature extractor to use the first convolution kernel to perform convolution feature extraction processing on the first image to obtain an extracted first feature image, the method further includes: Filling processing is performed on pixels at edge positions of the first image and the second image according to the convolution kernel of the largest size in the convolution kernel size set.

8. An image noise reduction device, characterized in that: include: An acquisition module is used to acquire a set of images to be denoised, wherein the set of images to be denoised includes multiple frames of noisy images obtained by photographing the same object in the same scene; An adjustment module, configured to adjust a feature extractor in a preset convolutional neural network according to a first image and a second image in the image set to be denoised, to obtain an adjusted target feature extractor, wherein the first image and the second image are different images, and the target feature extractor is configured to perform corresponding convolution feature extraction processing on the first image and the second image; A feature extraction module, used for calling the target feature extractor to perform convolution feature extraction processing on the first image and the second image to obtain an extracted target feature image; The denoising module is used to perform denoising on the target feature image to obtain a target image after denoising the first image.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images