Image denoising processing method and device thereof

Abstract

The application discloses an image denoising processing method and device, and belongs to the technical field of image processing. The method comprises the following steps: processing a first image based on a first processing mode to obtain a second image; processing at least one third image based on a second processing mode to obtain at least one fourth image, wherein the at least one third image is an image collected before the first image, and each fourth image corresponds to one third image; and performing joint denoising processing on the second image according to the at least one fourth image to obtain a fifth image subjected to joint denoising processing.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an image noise reduction processing method and apparatus. Background Technology

[0002] Currently, after a user shoots a video using an electronic device, the device can use image algorithms to reduce noise in the video to ensure smooth playback without flickering. In existing technologies, electronic devices can reduce the image size in the video, thereby enabling the use of high-performance and low-power image algorithms for noise reduction.

[0003] However, in the above noise reduction methods, although the electronic device uses a high-performance and low-power image algorithm to reduce the noise of the video, it also reduces the image size of the captured video, which results in lower clarity of the video image captured by the electronic device. Summary of the Invention

[0004] The purpose of this application is to provide an image noise reduction processing method, apparatus, electronic device, and readable storage medium that can solve the problem of low clarity in captured video images.

[0005] In a first aspect, embodiments of this application provide an image denoising processing method, which includes: processing a first image acquired based on a first processing method to obtain a second image; processing at least one frame of a third image based on a second processing method to obtain at least one frame of a fourth image, wherein the at least one frame of the third image is an image acquired before the first image, and each frame of the at least one frame of the fourth image corresponds to one frame of the third image; and performing joint denoising processing on the second image based on the at least one frame of the fourth image to obtain a fifth image after joint denoising processing.

[0006] Secondly, embodiments of this application provide an image denoising processing apparatus, which includes a processing module. The processing module is configured to process a first image acquired based on a first processing method to obtain a second image; and to process at least one frame of a third image based on the second processing method to obtain at least one frame of a fourth image, wherein the at least one frame of the third image is an image acquired before the first image, and each frame of the at least one fourth image corresponds to one frame of the third image; and to perform joint denoising processing on the second image based on the at least one frame of the fourth image to obtain a fifth image after joint denoising processing.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0011] In this embodiment of the application, the electronic device can process the first image acquired based on the first processing method to obtain the second image, and process at least one frame of image acquired before the first image (i.e. at least one frame of the third image) through the second processing method to obtain at least one frame of the fourth image, and then perform joint noise reduction processing on the second image through the at least one frame of the fourth image to obtain the noise-reduced fifth image. In this scheme, since the electronic device can use different processing methods on the currently acquired first image and at least one frame of images acquired before the first image, it can obtain image information different from that in the first image and at least one frame of the third image. That is, after different image processing, the first image and at least one frame of the third image result in a second image and at least one frame of the fourth image containing different image information. It can be understood that by processing the preceding and following frames differently, the electronic device simulates pixel shifting, thereby maximizing the preservation of image information within a continuous time period. Moreover, through multiple consecutive frames, the electronic device can stabilize the transition of temporal information on the one hand, and recover the image information of the first image acquired by the electronic device by utilizing the information differences between different frames on the other hand. This avoids the problem of low video image clarity that occurs when the electronic device reduces the image size of the captured video and uses high-performance and low-power image algorithms to perform noise reduction processing on the video. Thus, the clarity of the video captured by the electronic device is improved. Attached Figure Description

[0012] Figure 1 This is a schematic diagram illustrating an example of image downsampling in related technologies;

[0013] Figure 2This is a flowchart of an image noise reduction processing method provided in an embodiment of this application;

[0014] Figure 3 This is a schematic diagram illustrating an example of an image noise reduction processing method provided in an embodiment of this application;

[0015] Figure 4 This is a schematic diagram of the structure of an image noise reduction processing device provided in an embodiment of this application;

[0016] Figure 5 This is one of the hardware structure diagrams of an electronic device provided in the embodiments of this application;

[0017] Figure 6 This is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0020] The image noise reduction processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0021] Currently, users can perform increasingly more functions through electronic devices. For example, users can shoot videos using electronic devices. With the development of communication technology, users' demands for the image quality of videos shot by electronic devices are also gradually increasing. In related technologies, electronic devices can use image denoising algorithms to process the video footage. For example, electronic devices can reduce the size of the video image (i.e., reduce the video resolution) and then process the video image using high-performance and low-power image denoising algorithms to obtain a smooth, flicker-free video. However, compared to image downsampling in the RGB and YUV domains, downsampling on the Raw domain Bayer image array has a greater impact on image sharpness. Figure 1 As shown in (A), this is because the downsampling process in the RGB and YUV domains is as follows: the Bayer image is upsampled from a single-channel Bayer image to a three-channel image via Demosaic (i.e., image interpolation), and then downsampled to the target size; as shown in (A). Figure 1 As shown in (B), Bayer image downsampling first downsamples to the target size and then upsamples to a three-channel image via Demosaic. Obviously, the loss of sharpness from downsampling followed by upsampling is greater than that from upsampling followed by downsampling.

[0022] In this embodiment of the application, the electronic device processes the first image acquired based on the first processing method to obtain the second image, and processes at least one frame of image acquired before the first image (i.e. at least one frame of the third image) through the second processing method to obtain at least one frame of the fourth image. Then, the second image is subjected to joint noise reduction processing through the at least one frame of the fourth image to obtain the noise-reduced fifth image. In this scheme, because the electronic device can use different processing methods on the currently acquired first image and at least one frame of images acquired before the first image, it can obtain image information different from that in the first image and at least one frame of the third image. That is, after different image processing, the first image and at least one frame of the third image result in a second image and at least one frame of the fourth image containing different image information. It can be understood that by processing the preceding and following frames of images differently, the electronic device simulates pixel shifting, thereby maximizing the preservation of image information within a continuous time period. Moreover, through multiple consecutive frames of images, the electronic device can stabilize the transition of temporal information on the one hand, and recover the image information of the first image acquired by the electronic device by utilizing the information differences between different frames on the other hand. This avoids the problem of low video image clarity that occurs when the electronic device reduces the image size of the captured video and uses high-performance and low-power image algorithms to perform noise reduction processing on the video. Thus, the clarity of the video captured by the electronic device is improved.

[0023] This application provides an image noise reduction processing method. Figure 2A flowchart of an image noise reduction processing method provided in an embodiment of this application is shown. Figure 2 As shown, the image noise reduction processing method provided in this application embodiment may include the following steps 201 to 203.

[0024] Step 201: The electronic device processes the acquired first image based on the first processing method to obtain the second image.

[0025] In this embodiment of the application, when a video is captured and a first image is acquired, the electronic device uses a center interpolation processing method (i.e., the first processing method described above) to process the first image and obtain a second image.

[0026] In this embodiment of the application, the electronic device can acquire a first image of the original size through an image sensor, and then process the first image through center interpolation to obtain a second image of the second size.

[0027] It should be noted that the embodiments of this application address the issue of image sharpness loss in Bayer downsampling, and the first image mentioned above is a Bayer image.

[0028] It can be understood that the first image mentioned above is the current frame image captured by the electronic device during the video recording process. That is, the first image is not a fixed image, but changes according to the changes of the object being filmed by the electronic device until the electronic device finishes filming. The first image is then the last frame image of the video.

[0029] Optionally, in the embodiments of this application, the Bayer arrangement in the image sensor can be any of the following: RGGB, RYYB, RGBW or RGBIR, where IR is a filter.

[0030] Optionally, in the embodiments of this application, the above-mentioned filter can be any of the following: an infrared filter, an interference filter, or a Raman filter.

[0031] Optionally, in this embodiment, the original size is larger than the second size.

[0032] Optionally, in this embodiment of the application, when the shooting preview interface is displayed, the electronic device can receive the user's first input so that the electronic device can start shooting video.

[0033] Optionally, in this embodiment, the first input can be user input to the shooting control, user input to a physical button, or user voice input to the electronic device. Specifically, it can be determined according to actual usage needs, and this embodiment does not impose any limitations.

[0034] Specifically, the first input mentioned above can be a user's click input, long press input, swipe input, or preset trajectory input on the shooting control; or it can be a combination of physical buttons (such as the power button and volume buttons). The specific input can be determined according to actual usage needs, and this application embodiment does not impose any limitations.

[0035] Optionally, in the embodiments of this application, step 201 above can be specifically implemented by steps 201a to 201c below.

[0036] Step 201a: The electronic device performs interpolation processing on at least one first pixel point based on the actual coordinate information of at least one first pixel point in the first image to obtain the first coordinate information of at least one first pixel point.

[0037] In this embodiment of the application, the first coordinate information mentioned above is the coordinate information of the corresponding pixel after interpolation processing.

[0038] Specifically, the aforementioned first coordinate information is the coordinate information obtained by the electronic device after interpolating each of the at least one first pixel points.

[0039] In this embodiment of the application, the electronic device can perform interpolation processing on at least one first pixel point participating in the interpolation according to the target magnification and the first interpolation method to obtain the first coordinate information of at least one first pixel point.

[0040] It should be noted that, due to the special characteristics of Bayer arrays, the target multiplier is an integer power of 2 (e.g., 2, 4, 8).

[0041] Optionally, in this embodiment of the application, the target magnification can be preset by the user; or determined by the electronic device based on the size of the first image.

[0042] Optionally, the first interpolation method described above can be any of the following: nearest neighbor interpolation, bilinear interpolation, mean interpolation, or median interpolation.

[0043] For example, taking a target magnification of 2x as an example, and taking the first image as an RGGB arrangement as an example, such as Figure 3 As shown in (A), the electronic device interpolates a 4×4 Bayer image at a magnification of 2, and can extract the pixels corresponding to the R channel respectively. Figure 3 (represented by R1, R2, R3, and R4), G channel ( Figure 3 (Hereinafter referred to as G1) The corresponding pixel point ( Figure 3 (represented by G11, G12, G13, and G14), G channel ( Figure 3 The pixel corresponding to (hereinafter referred to as G2) Figure 3(represented by G21, G22, G23, and G24) and the corresponding pixels of the B channel ( Figure 3 (denoted by B1, B2, B3, and B4 in the original text). The electronic device can then use the pixel located between the pixels corresponding to the R channel, G1 channel, G2 channel, and B channel as the interpolated Bayer image, thus obtaining a 2×2 Bayer image; for example... Figure 3 As shown in (B), the electronic device interpolates the 4×4 Bayer image at a magnification of 2, and can extract the pixels corresponding to the R channel respectively. Figure 3 (represented by R1, R2, R3, and R4), G channel ( Figure 3 The pixel corresponding to G1 in the middle ( Figure 3 (represented by G11, G12, G13, and G14), G channel ( Figure 3 The pixel corresponding to (represented by G2) Figure 3 (represented by G21, G22, G23, and G24) and the corresponding pixels of the B channel ( Figure 3 (represented by B1, B2, B3, and B4). The electronic device can then perform mean processing on the three pixels located between the pixels corresponding to the R channel, the G1 channel, the G2 channel, and the B channel to obtain a mean-processed pixel for each channel. This mean-processed pixel for each channel is then used as the interpolated Bayer image, resulting in a 2×2 Bayer image.

[0044] It should be noted that, for the at least one first pixel participating in the interpolation and the coordinate information of the at least one first pixel participating in the interpolation, the electronic device can determine different at least one first pixel according to different interpolation algorithms, thereby obtaining the coordinate information corresponding to the at least one first pixel. That is, the number of pixels participating in the interpolation and the coordinate information are determined according to the interpolation algorithm, and can be determined according to actual usage requirements. This application embodiment does not impose any restrictions.

[0045] Step 201b: The electronic device obtains the equivalent center coordinate information of at least one first pixel based on the target reference magnification, the actual coordinate information of at least one first pixel, and at least one first weighting coefficient.

[0046] In this embodiment of the application, the first weighting coefficient is the ratio of the pixel value of the corresponding pixel point to the pixel value of the first image.

[0047] In this embodiment of the application, when the target reference magnification is determined, the electronic device can obtain the equivalent center coordinates of at least one first pixel point according to the equivalent center coordinate formula (1), the specific formula being:

[0048]

[0049] Where, ω i The first weighting coefficient, (x) i ,y i () represents the actual coordinates of at least one first pixel point involved in the interpolation. Let i be the equivalent center coordinates of at least one first pixel in the original image after interpolation, where i is the pixel number.

[0050] It should be noted that the above equivalent center coordinates are not the coordinates of pixels that actually exist in the first image, but are used to indicate that the phase of at least one first pixel does not change after interpolation.

[0051] Optionally, in this embodiment of the application, the target reference magnification can be preset by the user; or determined by the electronic device based on the first image size.

[0052] It should be noted that the target reference magnification when the electronic device obtains the equivalent center coordinate information of at least one first pixel is consistent with the target magnification when the electronic device performs interpolation processing. That is, the electronic device performs interpolation processing on at least one first pixel in the first image and obtains the center equivalent coordinates of at least one first pixel after interpolation at the same magnification.

[0053] Optionally, in this application, the aforementioned at least one first weighting coefficient can be preset by the user; or, the electronic device can determine it based on the ratio between the pixel value of the first image and the pixel value of at least one first pixel.

[0054] Optionally, in the embodiments of this application, the above pixel values ​​may include at least one of the following: pixel brightness value, pixel saturation value, pixel color temperature value, and pixel exposure value.

[0055] Step 201c: The electronic device determines the image region corresponding to N first pixels in the first image as the second image.

[0056] In this embodiment of the application, the above-mentioned N first pixels are pixels whose first coordinate information and equivalent center coordinate information satisfy a preset condition among at least one first pixel, and N is a positive integer.

[0057] In this embodiment of the application, the electronic device can input the coordinates of at least one first pixel into the equivalent central relation, thereby determining the image region corresponding to N first pixels that satisfy the central equivalence relation as the second image, and calculating it using formula (2) as follows:

[0058]

[0059] in, Let be the equivalent center coordinates of at least one first pixel in the original image after interpolation. Q represents the actual coordinates of at least one first pixel in the original image after interpolation, and Q is the target reference magnification.

[0060] It is understandable that the actual coordinates under the same reference magnification correspond to the equivalent center coordinates under the same reference magnification. In other words, if the reference magnification is different, the equivalent center coordinates corresponding to the actual coordinates will also be different.

[0061] In this embodiment, the electronic device obtains the equivalent center coordinates of at least one first pixel after interpolation using an equivalent center coordinate algorithm. This allows the image region corresponding to N first pixels among the at least one first pixel that satisfies the equivalent center relationship to be determined as the second image. In this way, the electronic device can ensure that the relative position of each pixel in the second image obtained after interpolation does not change with respect to the interpolated pixels in the first image, thereby improving the accuracy of image processing by the electronic device.

[0062] Step 202: The electronic device processes at least one frame of the third image based on the second processing method to obtain at least one frame of the fourth image.

[0063] In this embodiment of the application, the above-mentioned at least one third image is an image captured before the first image, and each fourth image in the at least one fourth image corresponds to a third image.

[0064] In this embodiment of the application, the electronic device may use an eccentric interpolation processing method (i.e. the second processing method described above) to process at least one frame of the third image to obtain at least one frame of the fourth image.

[0065] In this embodiment of the application, the electronic device can save at least one frame of the third image at its original size, and then process the at least one frame of the third image through an eccentric interpolation method to obtain at least one frame of the fourth image at the third size.

[0066] It should be noted that at least one of the above third images is a Bayer image.

[0067] Optionally, in the embodiments of this application, the aforementioned at least one third image may be an image captured by the electronic device before the first image; or, the aforementioned at least one third image may be all images captured by the electronic device before the first image.

[0068] Optionally, in the embodiments of this application, the third dimension is smaller than the original dimension, and the third dimension may be different from the second dimension.

[0069] Optionally, in the embodiments of this application, step 202 above can be specifically implemented by steps 202a to 202c below.

[0070] Step 202a: For each frame of the third image in at least one frame, the electronic device performs interpolation processing on at least one second pixel based on the actual coordinate information of at least one second pixel in the third image to obtain the second coordinate information of at least one second pixel.

[0071] In this embodiment of the application, the second coordinate information mentioned above is the coordinate information of the corresponding pixel after interpolation.

[0072] Specifically, the aforementioned second coordinate information is the coordinate information obtained by the electronic device after interpolating each of the at least one second pixel point.

[0073] In this embodiment of the application, the electronic device can perform interpolation processing on at least one second pixel point participating in the interpolation according to the target magnification and the first interpolation method to obtain the first coordinate information of at least one second pixel point.

[0074] It should be noted that, for the at least one second pixel point involved in the interpolation and the coordinate information of the at least one second pixel point involved in the interpolation, the electronic device can determine different at least one second pixel point according to different interpolation algorithms, thereby obtaining the coordinate information corresponding to the at least one second pixel point. That is, the number of pixels involved in the interpolation and the coordinate information are determined according to the interpolation algorithm, and can be determined according to actual usage requirements. This application embodiment does not impose any restrictions.

[0075] Step 202b: The electronic device obtains the equivalent center coordinate information of at least one second pixel based on the target reference magnification, the actual coordinate information of at least one second pixel, and at least one second weighting coefficient.

[0076] In this embodiment of the application, the second weighting coefficient is the ratio of the pixel value of the corresponding pixel point to the pixel value of a frame of the third image.

[0077] In this embodiment of the application, when the target reference magnification is determined, the electronic device can obtain the equivalent center coordinates of at least one first pixel point according to the center equivalent coordinate formula (3). The specific formula is as follows:

[0078]

[0079] Where, ω i The first weighting coefficient, (x) i ,y i () represents the actual coordinates of at least one second pixel point involved in the interpolation. Let i be the equivalent center coordinates of at least one second pixel in the original image after interpolation, where i is the pixel number.

[0080] Step 202c: The electronic device determines the image region corresponding to M second pixels in a third frame as a fourth frame.

[0081] In this embodiment of the application, the above-mentioned M second pixels are at least one second pixel whose second coordinate information and equivalent center coordinate information do not meet the preset conditions, and M is a positive integer.

[0082] In this embodiment of the application, the electronic device can input the coordinates of at least one second pixel into the central equivalence relation, thereby determining the image region corresponding to the M second pixels that do not satisfy the central equivalence relation as the fourth image, and calculating it using formula (4) as follows:

[0083]

[0084] in, Let be the equivalent center coordinates of at least one second pixel in the original image after interpolation. Q represents the actual coordinates of at least one second pixel in the original image after interpolation, and Q is the target reference magnification.

[0085] It should be noted that for each frame of the third image in at least one frame of the third image, the electronic device can perform the above steps 202a to 202c to obtain at least one frame of the fourth image. To avoid repetition, this will not be repeated here.

[0086] In this embodiment of the application, the electronic device obtains the equivalent center coordinates of at least one second pixel after interpolation using an equivalent center coordinate algorithm. This allows the image region corresponding to M first pixels among the at least one second pixel that does not satisfy the equivalent center relationship to be determined as a fourth frame image. In this way, the electronic device can make the relative position of each pixel in the fourth frame image obtained after interpolation change with respect to the interpolated pixels in the first image, simulating pixel shifting. Thus, the electronic device can retain image information to the maximum extent within a continuous time period.

[0087] Step 203: The electronic device performs joint noise reduction processing on the second image based on at least one frame of the fourth image to obtain the fifth image after joint noise reduction processing.

[0088] In this embodiment of the application, the electronic device can input at least one frame of the fourth image and the second image into the target neural network to obtain the fifth image after joint noise reduction processing.

[0089] Optionally, in this embodiment, the target neural network can be any of the following: a convolutional neural network, a generative adversarial neural network, or a periodic neural network. Specifically, it can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0090] Optionally, in the embodiments of this application, the aforementioned at least one fourth image can be the previous frame of the first image and an image processed by an eccentric interpolation method; or it can be all the previous frames of the first image and images processed by an eccentric interpolation method.

[0091] Optionally, in the embodiments of this application, step 203 can be implemented by step 203a as described below.

[0092] Step 203a: The electronic device adds pixel information from at least one frame of the fourth image to the second image based on the target processing method, to obtain the fifth image after joint noise reduction processing.

[0093] In this embodiment of the application, the target processing method is any one of the following: weighted method, convolution method, or high-pass filtering method.

[0094] It is understandable that at least one frame of the fourth image and the second image are obtained through different processing methods, so the image information in at least one frame of the fourth image and the second image is also different. Thus, the electronic device can use a target processing method to add the pixel information of at least one frame of the fourth image to the second image to obtain the fifth image after joint noise reduction processing.

[0095] Optionally, in this embodiment of the application, the electronic device can add the pixel information of at least one frame of the fourth image to the second image through target processing to obtain the RGB image of the fifth image after joint noise reduction processing.

[0096] This application provides an image noise reduction processing method. An electronic device can process a first image acquired based on a first processing method to obtain a second image, and process at least one frame of an image acquired before the first image (i.e., at least one frame of a third image) through a second processing method to obtain at least one frame of a fourth image. Then, the second image is subjected to joint noise reduction processing through the at least one frame of the fourth image to obtain a fifth image after noise reduction processing. In this scheme, since the electronic device can use different processing methods on the currently acquired first image and at least one frame of images acquired before the first image, it can obtain image information different from that in the first image and at least one frame of the third image. That is, after different image processing, the first image and at least one frame of the third image result in a second image and at least one frame of the fourth image containing different image information. It can be understood that by processing the preceding and following frames of images differently, the electronic device simulates pixel shifting, thereby maximizing the preservation of image information within a continuous time period. Moreover, through multiple consecutive frames of images, the electronic device can stabilize the transition of temporal information on the one hand, and recover the image information of the first image acquired by the electronic device by utilizing the information differences between different frames on the other hand. This avoids the problem of low video image clarity that occurs when the electronic device reduces the image size of the captured video and uses high-performance and low-power image algorithms for noise reduction processing. Thus, the clarity of the video captured by the electronic device is improved.

[0097] Optionally, in this embodiment of the application, before step 203 above, the image noise reduction processing provided in this embodiment of the application further includes the following steps 301 and 303, and step 203 above can be specifically implemented by step 203b below.

[0098] Step 301: The electronic device acquires the pixel value of each pixel in the second image and the pixel value of each pixel in the sixth image.

[0099] In this embodiment of the application, the sixth image is the previous frame of the first image and the image processed by the first processing method.

[0100] In this embodiment of the application, the electronic device can obtain the pixel value of each pixel in the second image by means of the correspondence between the coordinates and pixel values ​​of each pixel in the second image, and obtain the pixel value of each pixel in the sixth image by means of the correspondence between the coordinates and pixel values ​​of each pixel in the sixth image.

[0101] It should be noted that the above correspondence is preset by the user.

[0102] Step 302: The electronic device determines the image regions corresponding to pixels with the same pixel values ​​in the second image and the sixth image as similarity images between the second image and the sixth image based on the pixel values ​​of each pixel in the second image and the pixel values ​​of each pixel in the sixth image.

[0103] In this embodiment of the application, the electronic device can obtain the similarity image between the second image and the sixth image using the image similarity formula (5). The specific formula is as follows:

[0104]

[0105] Where clip(z,0,1) is the numerical constraint function, used to restrict the value to [0,1], k is the normalization coefficient, and f lowpass (·) represents the low-pass filter function; For the second image, This is the fifth image.

[0106] Specifically, the electronic device can acquire the pixel value of each pixel in the second image and the pixel value of each pixel in the sixth image, and then obtain a similarity image between the second image and the sixth image by means filtering.

[0107] Step 303: The electronic device performs weighted processing on the second image, the sixth image, and the similarity image to obtain the seventh image after temporal denoising.

[0108] In this embodiment of the application, the electronic device can perform weighted processing on the second image, the sixth image, and the similarity image using the weighting formula (6) to obtain the second image after temporal denoising. The specific formula is as follows:

[0109]

[0110] Where A(x, y) is the similarity image, For the second image, This is the sixth image.

[0111] Step 203b: The electronic device performs joint noise reduction processing on the seventh image after temporal domain noise reduction processing based on at least one frame of the fourth image to obtain the fifth image after joint noise reduction processing.

[0112] In this embodiment of the application, the seventh image after temporal denoising and at least one frame of the fourth image are input into the target neural network to obtain the fifth image after joint temporal denoising.

[0113] In this embodiment, since temporal denoising can eliminate noise jitter between frames, the resolution of the fifth image can be improved by denoising the seventh image after temporal denoising through at least one fourth image, thereby achieving the effect of super-resolution of image information. In this way, the clarity of the image processed by the electronic device is improved.

[0114] It should be noted that the image denoising processing apparatus method provided in this application embodiment can be executed by an image denoising processing apparatus, an electronic device, or a functional module or entity within an electronic device. This application embodiment uses the execution of the image denoising processing apparatus method by an image denoising processing apparatus as an example to illustrate the image denoising processing apparatus provided in this application embodiment.

[0115] Figure 4 A schematic diagram of a possible structure of the image noise reduction processing apparatus involved in an embodiment of this application is shown. For example... Figure 4 As shown, the image noise reduction processing device 70 may include a processing module 71.

[0116] The processing module 71 is configured to process the acquired first image based on a first processing method to obtain a second image; and process at least one frame of a third image based on a second processing method to obtain at least one frame of a fourth image, wherein the at least one frame of the third image is an image acquired before the first image, and each frame of the at least one frame of the fourth image corresponds to one frame of the third image; and perform joint noise reduction processing on the second image based on the at least one frame of the fourth image to obtain a fifth image after joint noise reduction processing.

[0117] In one possible implementation, the processing module 71 is specifically configured to: interpolate at least one first pixel based on the actual coordinate information of at least one first pixel in the first image to obtain first coordinate information of at least one first pixel, wherein the first coordinate information is the coordinate information of the corresponding first pixel after interpolation; and obtain equivalent center coordinate information of at least one first pixel based on the target reference magnification, the actual coordinate information of at least one first pixel, and at least one first weighting coefficient, wherein the first weighting coefficient is the ratio of the pixel value of the corresponding pixel to the pixel value of the first image; and determine the image region corresponding to N first pixels in the first image as the second image, wherein the N first pixels are pixels whose first coordinate information and equivalent center coordinate information satisfy a preset condition, and N is a positive integer.

[0118] In one possible implementation, the processing module 71 is specifically configured to, for each frame of at least one third image, perform interpolation processing on at least one second pixel based on the actual coordinate information of at least one second pixel in the third image to obtain the second coordinate information of at least one second pixel, wherein the second coordinate information is the coordinate information of the corresponding pixel after interpolation processing; and obtain the equivalent center coordinate information of at least one second pixel based on the target reference magnification, the actual coordinate information of at least one second pixel, and at least one second weighting coefficient, wherein the second weighting coefficient is the ratio of the corresponding pixel value to the pixel value of the third image; and determine the image region corresponding to M second pixels in the third image as a fourth image, wherein the M second pixels are the pixels whose second coordinate information and equivalent center coordinate information do not meet the preset conditions, and M is a positive integer.

[0119] In one possible implementation, the image denoising processing apparatus provided in this application embodiment further includes: an acquisition module and a determination module. The acquisition module is used to acquire the pixel value of each pixel in the second image and the pixel value of each pixel in a sixth image before the processing module performs joint denoising processing on the second image based on at least one frame of the fourth image to obtain a fifth image after joint denoising processing. The sixth image is the previous frame of the first image and is an image processed by the first processing method. The determination module is used to determine the image regions corresponding to pixels with the same pixel values ​​in the second image and the sixth image as similarity images between the second image and the sixth image, based on the pixel values ​​of each pixel in the second image and the pixel values ​​of each pixel in the sixth image acquired by the acquisition module. The aforementioned processing module 71 is further used to perform weighted processing on the second image, the sixth image, and the similarity image to obtain a seventh image after temporal denoising processing; specifically, the aforementioned processing module 71 is used to perform joint denoising processing on the seventh image after temporal denoising processing based on at least one frame of the fourth image to obtain a fifth image after joint denoising processing.

[0120] In one possible implementation, the processing module is specifically used to add pixel information of at least one frame of the fourth image to the second image based on the target processing method to obtain the fifth image after joint noise reduction processing. The target processing method is any one of the following: weighted method, convolution method, or high-pass filtering method.

[0121] This application provides an image noise reduction processing device. Since the image noise reduction processing can apply different processing methods to the currently acquired first image and at least one frame of images acquired before the first image, it can obtain image information different from that in the first image and at least one frame of third image. That is, after different image processing, the first image and at least one frame of third image are the second image and at least one frame of fourth image, which contain different image information. It can be understood that by applying different processing methods to the preceding and following frames of images, the image noise reduction processing device simulates pixel shifting, thereby maximizing the preservation of image information within a continuous time period. Moreover, through multiple consecutive frames of images, the electronic device can stabilize the transition of temporal information on the one hand, and recover the image information of the first image acquired by the image noise reduction processing device by utilizing the information differences between different frames on the other hand. This avoids the problem of low video image clarity that occurs when the image noise reduction processing device reduces the image size of the captured video and uses high-performance and low-power image algorithms to perform noise reduction processing on the video. Thus, the clarity of the video captured by the image noise reduction processing device is improved.

[0122] The image noise reduction processing device in this application embodiment can be a device, or a component, integrated circuit, or chip in an electronic device. The device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0123] The image noise reduction processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0124] The image noise reduction processing device provided in this application embodiment can achieve... Figures 1 to 3The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0125] Optionally, such as Figure 5 As shown, this application embodiment also provides an electronic device 90, including a processor 91 and a memory 92. The memory 92 stores a program or instructions that can run on the processor 91. When the program or instructions are executed by the processor 91, they implement the various steps of the above-described image noise reduction processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0126] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0127] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0128] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0129] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0130] The processor 110 is configured to process the acquired first image based on a first processing method to obtain a second image; and process at least one frame of a third image based on a second processing method to obtain at least one frame of a fourth image, wherein the at least one frame of the third image is an image acquired before the first image, and each frame of the at least one frame of the fourth image corresponds to one frame of the third image; and perform joint noise reduction processing on the second image based on the at least one frame of the fourth image to obtain a fifth image after joint noise reduction processing.

[0131] This application provides an electronic device that can apply different processing methods to the currently acquired first image and at least one frame of images acquired before the first image. This allows the electronic device to obtain image information in the first image that is different from that in at least one frame of third image. In other words, after different image processing, the first image and at least one frame of third image result in a second image and at least one frame of fourth image containing different image information. It can be understood that by applying different processing methods to the preceding and following frames, the electronic device simulates pixel shifting, thereby maximizing the preservation of image information within a continuous time period. Moreover, through multiple consecutive frames of images, the electronic device can stabilize the transition of temporal information and recover the image information of the first image acquired by the electronic device by utilizing the information differences between different frames. This avoids the problem of low video image clarity that occurs when the electronic device reduces the image size of the captured video and uses high-performance and low-power image algorithms for noise reduction processing. Thus, the clarity of the video captured by the electronic device is improved.

[0132] Optionally, in this embodiment, the processor 110 is specifically configured to: perform interpolation processing on at least one first pixel point according to the actual coordinate information of at least one first pixel point in the first image to obtain the first coordinate information of at least one first pixel point, wherein the first coordinate information is the coordinate information of the corresponding pixel point after interpolation processing; and obtain the equivalent center coordinate information of at least one first pixel point according to the target reference magnification, the actual coordinate information of at least one first pixel point and at least one first weight coefficient, wherein the first weight coefficient is the ratio of the pixel value of the corresponding pixel point to the pixel value of the first image; and determine the image region corresponding to N first pixel points in the first image as the second image, wherein the N first pixel points are the pixel points in the at least one first pixel point whose first coordinate information and equivalent center coordinate information satisfy a preset condition, and N is a positive integer.

[0133] Optionally, in this embodiment, the processor 110 is specifically configured to, for each frame of at least one third image, perform interpolation processing on at least one second pixel based on the actual coordinate information of at least one second pixel in the frame of the third image to obtain the second coordinate information of at least one second pixel, wherein the second coordinate information is the coordinate information of the corresponding pixel after interpolation processing; and obtain the equivalent center coordinate information of at least one second pixel based on the target reference magnification, the actual coordinate information of at least one second pixel, and at least one second weighting coefficient, wherein the second weighting coefficient is the ratio of the pixel value of the corresponding pixel to the pixel value of the frame of the third image; and determine the image region corresponding to M second pixels in the frame of the third image as a frame of the fourth image, wherein the M second pixels are the pixels whose second coordinate information and equivalent center coordinate information do not meet the preset conditions, and M is a positive integer.

[0134] Optionally, in this embodiment, the processor 110 is further configured to: obtain the pixel value of each pixel in the second image and the pixel value of each pixel in the sixth image, wherein the sixth image is the previous frame of the first image and has been processed by the first processing method; and determine the image regions corresponding to pixels with the same pixel values ​​in the second and sixth images as similarity images based on the pixel values ​​of each pixel in the second and sixth images; and perform weighted processing on the second image, the sixth image, and the similarity image to obtain a seventh image after temporal denoising. Specifically, the processor 110 is configured to perform joint denoising processing on the seventh image after temporal denoising processing based on at least one frame of the fourth image to obtain the fifth image after joint denoising processing.

[0135] Optionally, in this embodiment, the processor 110 is specifically used to add the pixel information of at least one frame of the fourth image to the second image using a target processing method to obtain the fifth image after joint noise reduction processing. The target processing method is any one of the following: weighted method, convolution method, or high-pass filtering method.

[0136] The electronic device provided in this application embodiment can implement the various processes implemented in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0137] For details on the beneficial effects of the various implementation methods in this embodiment, please refer to the beneficial effects of the corresponding implementation methods in the above method embodiments. To avoid repetition, these will not be repeated here.

[0138] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0139] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0140] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0141] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0142] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0143] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0144] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0145] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the image noise reduction processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0146] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0148] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image denoising processing method, characterized by, The method comprises: processing the collected first image based on a first processing mode to obtain a second image; processing at least one third image based on a second processing mode to obtain at least one fourth image, the at least one third image being an image collected before the first image, and each fourth image corresponding to a third image; performing joint noise reduction processing on the second image according to the at least one fourth image to obtain a fifth image after joint noise reduction processing; the processing of the collected first image based on the first processing mode to obtain the second image comprises: performing interpolation processing on at least one first pixel point in the first image according to actual coordinate information of the at least one first pixel point to obtain first coordinate information of the at least one first pixel point, the first coordinate information being coordinate information after interpolation processing of the corresponding pixel point; obtaining equivalent center coordinate information of the at least one first pixel point according to a target reference magnification, the actual coordinate information of the at least one first pixel point, and at least one first weight coefficient, the first weight coefficient being a proportion of a pixel value of the corresponding pixel point to a pixel value of the first image; determining an image region corresponding to N first pixel points in the first image as the second image, the N first pixel points being pixel points in the at least one first pixel point whose first coordinate information and equivalent center coordinate information satisfy a preset condition, and N being a positive integer.

2. The method of claim 1, wherein, the processing of the at least one third image based on the second processing mode to obtain the at least one fourth image comprises: for each third image in the at least one third image, performing interpolation processing on at least one second pixel point in the third image according to actual coordinate information of the at least one second pixel point to obtain second coordinate information of the at least one second pixel point, the second coordinate information being coordinate information after interpolation processing of the corresponding pixel point; obtaining equivalent center coordinate information of the at least one second pixel point according to a target reference magnification, the actual coordinate information of the at least one second pixel point, and at least one second weight coefficient, the second weight coefficient being a proportion of a pixel value of the corresponding pixel point to a pixel value of the third image; determining an image region corresponding to M second pixel points in the third image as a fourth image, the M second pixel points being pixel points in the at least one second pixel point whose second coordinate information and equivalent center coordinate information do not satisfy the preset condition, and M being a positive integer.

3. The method of claim 1, wherein, before the joint noise reduction processing on the second image according to the at least one fourth image to obtain the fifth image after joint noise reduction processing, the method further comprises: obtaining a pixel value of each pixel point in the second image and a pixel value of each pixel point in a sixth image, the sixth image being a previous frame of the first image and being an image processed by the first processing mode; According to the pixel value of each pixel point in the second image and the pixel value of each pixel point in the sixth image, an image region corresponding to the pixel points with the same pixel value in the second image and the sixth image is determined as a similarity image of the second image and the sixth image; The second image, the sixth image and the similarity image are weighted to obtain a seventh image after time domain noise reduction processing; The joint noise reduction processing of the second image according to the at least one fourth image comprises: The joint noise reduction processing of the seventh image after time domain noise reduction processing according to the at least one fourth image comprises:

4. The method according to claim 1 or 3, characterized in that, The joint noise reduction processing of the second image according to the at least one fourth image comprises: The pixel point information of the at least one fourth image is added to the second image based on a target processing mode to obtain the fifth image after joint noise reduction processing, and the target processing mode is any one of the following: a weighting mode, a convolution mode and a high-pass filtering mode.

5. An image noise reduction processing apparatus characterized by comprising: The image noise reduction processing device comprises a processing module; The processing module is configured to process a first image collected based on a first processing mode to obtain a second image, process at least one third image based on a second processing mode to obtain at least one fourth image, and perform joint noise reduction processing of the second image according to the at least one fourth image to obtain a fifth image after joint noise reduction processing, wherein the at least one third image is an image collected before the first image, each fourth image corresponds to a third image, and the first processing mode and the second processing mode are different. The processing module is specifically configured to perform interpolation processing on at least one first pixel point in the first image according to actual coordinate information of the at least one first pixel point to obtain first coordinate information of the at least one first pixel point, the first coordinate information is coordinate information after interpolation processing of a corresponding pixel point, obtain equivalent center coordinate information of the at least one first pixel point according to a target reference magnification, the actual coordinate information of the at least one first pixel point and at least one first weight coefficient, the first weight coefficient is a proportion of a pixel value of a corresponding pixel point to a pixel value of the first image, and determine an image region corresponding to N first pixel points in the first image as the second image, the N first pixel points are pixel points whose first coordinate information and equivalent center coordinate information satisfy a preset condition in the at least one first pixel point, and N is a positive integer.

6. The apparatus of claim 5, wherein, The processing module is specifically configured to perform interpolation processing on at least one second pixel point in each third image according to actual coordinate information of the at least one second pixel point in the third image to obtain second coordinate information of the at least one second pixel point, and the second coordinate information is coordinate information after interpolation processing of a corresponding pixel point. According to the target reference magnification, the actual coordinate information of the at least one second pixel point, and at least one second weight coefficient, equivalent center coordinate information of the at least one second pixel point is obtained, the second weight coefficient is a proportion of a pixel value of a corresponding pixel point to a pixel value of the one frame of third image; And the image region corresponding to the M second pixel points in the one frame of third image is determined as one frame of fourth image, the M second pixel points are the pixel points whose second coordinate information and equivalent center coordinate information do not satisfy the preset condition in the at least one second pixel point, and M is a positive integer.

7. The apparatus of claim 5, wherein, The image noise reduction processing apparatus further comprises an acquisition module and a determination module. The acquisition module is configured to acquire a pixel value of each pixel point in the second image and a pixel value of each pixel point in a sixth image before the processing module performs joint noise reduction processing on the second image according to the at least one frame of fourth image to obtain a fifth image, the sixth image being a previous frame of the first image and being an image processed by the first processing mode. The determination module is configured to determine, according to the pixel value of each pixel point in the second image and the pixel value of each pixel point in the sixth image acquired by the acquisition module, an image region corresponding to the pixel points with the same pixel value in the second image and the sixth image as a similarity image of the second image and the sixth image. The processing module is further configured to perform weighted processing on the second image, the sixth image, and the similarity image to obtain a seventh image processed in time domain. The processing module is specifically configured to perform joint noise reduction processing on the seventh image processed in time domain according to the at least one frame of fourth image to obtain the fifth image processed by joint noise reduction.

8. The apparatus of claim 5 or 7, wherein, The processing module is specifically configured to add pixel point information of the at least one frame of fourth image to the second image based on a target processing mode to obtain the fifth image processed by joint noise reduction, the target processing mode being any one of a weighted mode, a convolution mode, and a high-pass filtering mode.

Images