An image denoising method, device, storage medium and product

Abstract

The application discloses an image denoising method and device, a storage medium and a product. The method comprises the following steps: acquiring a current frame image and a next frame image; the exposure time of the current frame image and the next frame image is equal; performing denoising processing on the current frame image by using the next frame image to obtain a first denoised image of the current frame; performing motion estimation and compensation on the first denoised image of the current frame and a reference frame image of a previous frame to obtain a reference frame image of the current frame; and displaying the reference frame image of the current frame based on the reference frame image of the current frame. Thus, the current frame image is first denoised by using the next frame image to obtain the first denoised image of the current frame, and then the first denoised image of the current frame is denoised again by using the reference frame image of the previous frame, so that the noise of the finally obtained reference frame image of the current frame is greatly reduced, that is, the denoising effect is improved, and the definition of the reference frame image of the current frame is improved.

Description

Technical Field

[0001] This application relates to denoising technology, and more particularly to an image denoising method, apparatus, storage medium, and product. Background Technology

[0002] To ensure the quality of video and preview images, denoising is typically performed using a denoising module in the Image Signal Processing (ISP) chain. In recent years, some manufacturers have attempted to achieve better denoising results by using a reference frame from the previous frame to denoise the current frame before displaying it; however, this method of denoising is not very effective. Summary of the Invention

[0003] This application provides an image denoising method, apparatus, storage medium, and product.

[0004] The technical solution of this application is implemented as follows:

[0005] Firstly, an image denoising method is provided, the method comprising:

[0006] Acquire the current frame image and the next frame image; wherein the exposure duration of the current frame image and the next frame image is equal;

[0007] The current frame image is denoised using the next frame image to obtain the first denoised image of the current frame;

[0008] Motion estimation and compensation are performed on the first denoised image of the current frame and the obtained reference frame image of the previous frame to obtain the reference frame image of the current frame;

[0009] The image is displayed based on a reference frame image of the current frame.

[0010] Secondly, an image denoising apparatus is provided, the apparatus comprising:

[0011] An acquisition unit is used to acquire the current frame image and the next frame image; wherein the exposure duration of the current frame image and the next frame image is equal;

[0012] The processing unit is configured to denoise the current frame image using the next frame image to obtain a first denoised image of the current frame; to perform motion estimation and compensation on the first denoised image of the current frame and the obtained reference frame image of the previous frame to obtain a reference frame image of the current frame; and to display the reference frame image of the current frame.

[0013] Thirdly, an electronic device is provided, comprising: a processor and a memory configured to store a computer program capable of running on the processor, wherein the processor is configured to perform the steps of the method of the first aspect when running the computer program.

[0014] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of the first aspect.

[0015] Fifthly, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the method of the first aspect.

[0016] This application provides an image denoising method, apparatus, storage medium, and product. The method first performs denoising processing on the current frame image using the next frame image to obtain a first denoised image of the current frame. Then, it performs denoising processing on the first denoised image of the current frame again using a reference frame image from the previous frame. This significantly reduces the noise in the final reference frame image of the current frame, thereby improving the denoising effect and enhancing the clarity of the reference frame image of the current frame. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 1 ;

[0018] Figure 2 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 2 ;

[0019] Figure 3 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 3 ;

[0020] Figure 4 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 4 ;

[0021] Figure 5 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 5 ;

[0022] Figure 6 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 6 ;

[0023] Figure 7 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 7 ;

[0024] Figure 8 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 8 ;

[0025] Figure 9 This is a schematic diagram of the structure of the image denoising device in the embodiments of this application;

[0026] Figure 10 This is a schematic diagram of the structure of the electronic equipment in the embodiments of this application. Detailed Implementation

[0027] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing the embodiments only and is not intended to limit the scope of this application.

[0029] In the following description, references to "some embodiments," "this embodiment," "this embodiment," and examples, etc., describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0030] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0031] In this embodiment, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0032] This application provides an image denoising method. Figure 1 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 1 It is applied to electronic devices, such as smartphones, tablets, and cameras. For example... Figure 1 As shown, the image denoising method may include:

[0033] S101: Obtain the current frame image and the next frame image; wherein the exposure time of the current frame image and the next frame image are equal.

[0034] The current frame refers to the frame of image to be processed. The next frame refers to the frame of image that immediately follows the current frame.

[0035] Exposure time refers to the length of time a camera or image sensor keeps its shutter open while capturing an image. The exposure time for the current frame and the next frame must be equal; that is, the shutter time for the camera or image sensor to capture the current frame and the next frame must be the same. This helps maintain the consistency and stability of the video.

[0036] S102: Use the next frame image to denoise the current frame image to obtain the first denoised image of the current frame.

[0037] By using the next frame image to denoise the current frame image, the noise in the current frame image is reduced, and the clarity and quality of the first denoised image of the current frame after denoising is improved.

[0038] S103: Perform motion estimation and compensation on the first denoised image of the current frame and the obtained reference frame image of the previous frame to obtain the reference frame image of the current frame.

[0039] Since the first denoised image of the current frame includes information from the next frame, the reference frame image of the previous frame can be understood as an image that includes information from the previous frame and all previous frames, as well as information from the current frame. The reference frame image of the current frame can be understood as an image that includes information from the current frame and all previous frames, as well as information from the next frame.

[0040] The reference frame image of the previous frame is stored in a preset storage space. Therefore, when S103 is executed, the reference frame image of the previous frame of the current frame is first obtained from the preset storage space, and then motion estimation and compensation are performed on the first denoised image of the current frame and the reference frame image of the previous frame to obtain the reference frame image of the current frame.

[0041] In some embodiments, after performing S103, the method further includes: storing the reference frame image of the current frame to a preset storage space.

[0042] Here, the reference frame image of the current frame is stored in a preset storage space, so that the reference frame image of the current frame can be used to process the image of the next frame. The preset storage space can be a double data rate (DDR) storage space.

[0043] S104: Display the reference frame image based on the current frame.

[0044] In some embodiments, the reference frame image of the current frame is displayed directly.

[0045] In other embodiments, S104 may include:

[0046] Perform image processing operations on the reference frame image of the current frame to obtain the processed reference frame image;

[0047] Displays the processed reference frame image.

[0048] The image processing operations here include at least: image stabilization, distortion correction, noise reduction, enhancement, scaling, etc.

[0049] It should be noted that when the current frame image is the first frame image, the reference frame image of the previous frame is empty.

[0050] Here, the execution entity of S101 to S104 can be the processor of an electronic device.

[0051] In this embodiment, the current frame image is first denoised using the next frame image to obtain the first denoised image of the current frame. Then, the first denoised image of the current frame is denoised again using the reference frame image of the previous frame, so that the noise of the final reference frame image of the current frame is greatly reduced, that is, the denoising effect is improved and the clarity of the reference frame image of the current frame is improved.

[0052] When the image sensor built into the electronic device sequentially outputs the current frame image and the next frame image in Raw format, in some embodiments, S102 may include Figure 2 The steps shown are as follows:

[0053] S201: Align the next frame image in Raw format with the current frame image in Raw format to obtain the first alignment parameter.

[0054] Raw format is an unprocessed, uncompressed digital image format.

[0055] Regarding alignment operations, the specific steps can be:

[0056] First, feature points are extracted from the current frame and the next frame. Feature points are regions in an image that have salient features, such as corners, edges, and textures. This can be done using feature detection algorithms such as SIFT, SURF, and ORB.

[0057] Next, feature points in the current frame are matched with feature points in the next frame. This typically involves calculating the similarity or distance between feature points and finding the best matching pair. Feature matching algorithms can include brute-force matching, FLANN matching, etc.

[0058] Next, based on the matched feature point pairs, a transformation model can be estimated to describe the geometric transformation from the current frame to the next frame. This transformation model can be an affine transformation matrix, a homography matrix, or other types of transformation model.

[0059] Next, the estimated transformation model is used to transform the subsequent frame image into the coordinate system of the current frame. This typically involves pixel interpolation and resampling to ensure that the transformed image is aligned with the current frame image.

[0060] Next, after applying the transformation model, it is important to evaluate the alignment quality. This can be done by comparing the similarity between the aligned image and the current frame image, calculating the alignment error, or using other evaluation metrics. If the alignment quality does not meet the requirements, the parameters of the feature extraction, feature matching, or transformation model can be adjusted, and the above steps can be repeated.

[0061] Finally, based on the transformation model applied, the first alignment parameters can be obtained. These parameters describe the geometric transformations from the current frame to the next frame, including rotation, translation, scaling, etc.

[0062] S202: Convert the current frame image from Raw format to RGB format, and convert the next frame image from Raw format to RGB format.

[0063] RGB format is a method of representing color that uses the brightness of three primary colors of light—red, green, and blue—to create various colors. In RGB format, the color of each pixel is represented by the brightness values ​​of the red, green, and blue channels. The brightness value ranges from 0 to 255, where 0 represents no light and 255 represents maximum brightness. By adjusting the brightness value of each channel, various colors can be created. For example, red can be represented as (255, 0, 0), indicating that the brightness value of the red channel is 255, while the brightness values ​​of the green and blue channels are both 0.

[0064] S203: Based on the first alignment parameter, align the next frame image in RGB format to the current frame image in RGB format to obtain a new next frame image in RGB format.

[0065] That is, after the alignment operation, the new frame image in RGB format is aligned with the current frame image in RGB format.

[0066] S204: Use the new next frame image in RGB format to denoise the current frame image in RGB format, and obtain the first denoised image of the current frame in RGB format.

[0067] The denoising process of the current frame image in RGB format is performed using a new subsequent frame image in RGB format, which reduces noise in the current frame image in RGB format and improves the clarity and quality of the first denoised image of the current frame in RGB format after denoising.

[0068] Furthermore, in some embodiments, S103 may include Figure 3 The steps shown are as follows:

[0069] S301: Convert the first denoised image of the current frame from RGB format to YUV format.

[0070] The YUV format separates the luminance (Y) and color (U and V) information of an image. "Y" represents luminance, which is the grayscale value; while "U" and "V" represent chrominance, which describe the color and saturation of the image and are used to specify the color of a pixel.

[0071] S302: Align the reference frame image of the previous frame in YUV format with the first denoised image of the current frame in YUV format to obtain a new reference frame image of the previous frame in YUV format.

[0072] That is, after the alignment operation, the new reference frame image of the previous frame in YUV format is aligned with the first denoised image of the current frame in YUV format.

[0073] S303: Perform motion estimation and compensation on the first denoised image of the current frame in YUV format and the new reference frame image of the previous frame in YUV format to obtain the reference frame image of the current frame in YUV format.

[0074] Here, motion estimation and compensation operations are also a way to denoise.

[0075] That is, the first denoised image of the current frame in YUV format is denoised again using the new reference frame image of the previous frame in YUV format, so that the noise of the reference frame image of the current frame in YUV format is reduced and the clarity of the reference frame image of the current frame is improved.

[0076] Based on the above embodiments, this application provides an example of an image denoising method. Figure 4 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 4 ,like Figure 4 As shown, image denoising methods may include:

[0077] S401: The sensor sequentially outputs the initial current frame image and the initial next frame image in Raw format. After processing by the Raw statistics module, the processed current frame image and next frame image are obtained and stored in DDR.

[0078] The Raw statistics module is typically used to process images in Raw format. Raw format is an unprocessed raw image data format that retains pixel value information directly output from the image sensor without post-processing such as color space conversion, noise reduction, or white balance adjustment. The Raw statistics module typically includes the following processing operations: color space conversion, noise reduction, and white balance adjustment.

[0079] S402: Obtain the current frame image and the next frame image in Raw format from DDR, align the next frame image in Raw format to the current frame image in Raw format, and obtain the first alignment parameter.

[0080] S403: Obtain the current frame image in Raw format from DDR, process the current frame image in Raw format through the Raw domain processing module, and obtain the current frame image in RGB format.

[0081] S404: Obtain the next frame image in Raw format from DDR, and use the next frame image in Raw format and the first alignment parameter as input to the RGB domain alignment module to obtain a new next frame image in Raw format.

[0082] Specifically, the next frame image in Raw format is converted to the next frame image in RGB format, and then, based on the first alignment parameter, the next frame image in RGB format is aligned with the current frame image in RGB format to obtain a new next frame image in Raw format.

[0083] S405: The new next frame image in RGB format and the current frame image in RGB format are processed by the RGB domain High Dynamic Range (HDR) module to obtain the first denoised image of the current frame in RGB format.

[0084] Here, the RGB domain HDR module is used to denoise the current frame image in RGB format using a new subsequent frame image in RGB format, resulting in the first denoised image of the current frame in RGB format. Specifically, the RGB domain HDR module can be used as a multi-frame noise reduction (MFNR) module based on a finite impulse response filter (FIR) for noise reduction.

[0085] S406: The first denoised image of the current frame in RGB format is processed sequentially through the RGB domain module and the spatial domain conversion module to obtain the second denoised image of the current frame in YUV format, and stored in DDR.

[0086] The RGB domain module includes at least the following processing operations: color correction, gamma correction, etc.

[0087] The spatial domain conversion module is used to convert between RGB and YUV formats.

[0088] S407: Obtain the reference frame image of the previous frame in YUV format from DDR, process the reference frame image of the previous frame in YUV format through the YUV domain alignment module, and obtain a new reference frame image of the previous frame that is aligned with the second denoised image of the current frame.

[0089] S408: After processing the second denoised image of the current frame in YUV format and the new reference frame image of the previous frame in YUV format through the Motion Compensation Temporal Filter (MCTF) module, the reference frame image of the current frame in YUV format is obtained and stored in DDR.

[0090] MCTF is a YUV domain time-domain denoising algorithm that uses the Infinite Impulse Response (IIR) method.

[0091] S409: The reference frame image of the current frame in YUV format is processed by the YUV domain module to obtain the processed reference frame image, and stored in DDR for use in video recording and preview.

[0092] The YUV domain module includes at least the following processing operations: image stabilization, distortion correction, noise reduction, enhancement, and scaling.

[0093] Among them, S407 to S409 fall under the category of Baseband Processing (BE). This application has no impact on MCTF performance on BE and can meet the requirements of 4K@60fps, thereby improving the signal-to-noise ratio and detail in this scene.

[0094] Thus, this application uses the RGB domain YUV module as an FIR-based RGB domain temporal denoising module, which can improve the signal-to-noise ratio of the current frame image by up to 3dB, that is, improve the input signal-to-noise ratio of MCTF, thereby further improving the denoising effect of MCTF.

[0095] When the HDR sensor outputs long-frame and short-frame images (Raw format) with different exposure durations at the same time, the long-frame and short-frame images are first fused in the RGB domain, and then the following steps are performed.

[0096] In some embodiments, S102 may include Figure 5 The steps shown are as follows:

[0097] S501: Align the next frame image in RGB format with the current frame image in RGB format to obtain the second alignment parameter.

[0098] In some embodiments, before executing S501, the method further includes: acquiring a long frame image and a short frame image in Raw format at the current moment, and acquiring a long frame image and a short frame image in Raw format at the next moment; performing high dynamic range fusion processing on the long frame image and the short frame image in Raw format at the current moment in the RGB domain to obtain a current frame image in RGB format; and performing high dynamic range fusion processing on the long frame image and the short frame image in Raw format at the next moment in the RGB domain to obtain a next frame image in RGB format.

[0099] The exposure time for a long frame image is longer than that for a short frame image.

[0100] In some embodiments, in the RGB domain, high dynamic range fusion processing is performed on the current time-varying long-frame image and short-frame image in Raw format to obtain the current frame image in RGB format. This includes: aligning the long-frame image in Raw format to the short-frame image to obtain a third alignment parameter; converting the long-frame image from Raw format to RGB format and converting the short-frame image from Raw format to RGB format; aligning the long-frame image in RGB format to the short-frame image based on the third alignment parameter to obtain a new long-frame image in RGB format; and using the new long-frame image in RGB format to denoise the short-frame image in RGB format to obtain the current frame image in RGB format.

[0101] It's important to note that long-frame images, due to their longer exposure times, typically contain more information. This information may be lost when converted to short-frame images because the shorter exposure times of short-frame images may not capture all the details found in the long-frame images. Therefore, aligning a long-frame image to a short-frame image can better preserve this information.

[0102] Accordingly, in some embodiments, in the RGB domain, high dynamic range fusion processing is performed on the long frame image and short frame image in Raw format at the next time step to obtain the next frame image in RGB format. This includes: aligning the long frame image in Raw format to the short frame image to obtain a fourth alignment parameter; converting the long frame image from Raw format to RGB format, and converting the short frame image from Raw format to RGB format; aligning the long frame image in RGB format to the short frame image based on the fourth alignment parameter to obtain a new long frame image in RGB format; and using the new long frame image in RGB format to denoise the short frame image in RGB format to obtain the next frame image in RGB format.

[0103] S502: Convert the current frame image from RGB format to YUV format, and convert the next frame image from RGB format to YUV format.

[0104] S503: Based on the second alignment parameter, align the next frame image in YUV format to the current frame image in YUV format to obtain a new next frame image in YUV format.

[0105] That is, after the alignment operation, the new next frame image in YUV format is aligned with the current frame image in YUV format.

[0106] S504: Use the new next frame image in YUV format to denoise the current frame image in YUV format, and obtain the first denoised image of the current frame in YUV format.

[0107] The current frame image in YUV format is denoised using a new subsequent frame image in YUV format, which reduces noise in the current frame image in YUV format and improves the clarity and quality of the first denoised image of the current frame in YUV format after denoising.

[0108] Further, in some embodiments, S103 may include: aligning the reference frame image of the previous frame in YUV format to the first denoised image of the current frame in YUV format to obtain a new reference frame image of the previous frame in YUV format; performing motion estimation and compensation on the first denoised image of the current frame in YUV format and the new reference frame image of the previous frame in YUV format to obtain a reference frame image of the current frame in YUV format.

[0109] Based on the above embodiments, this application provides an example of an image denoising method. Figure 6 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 6 ,like Figure 6 As shown, image denoising methods may include:

[0110] S601: The HDR sensor outputs the initial long frame image and initial short frame image in Raw format at the current moment. After processing by the Raw statistics module, the processed long frame image and short frame image at the current moment are obtained and stored in DDR. The HDR sensor outputs the initial long frame image and initial short frame image in Raw format at the next moment. After processing by the Raw statistics module, the processed long frame image and short frame image at the next moment are obtained and stored in DDR.

[0111] The Raw statistics module is typically used to process images in Raw format. Raw format is an unprocessed raw image data format that retains pixel value information directly output from the image sensor without post-processing such as color space conversion, noise reduction, or white balance adjustment. The Raw statistics module typically includes the following processing operations: color space conversion, noise reduction, and white balance adjustment.

[0112] S602: Obtain the current time-lapse long frame image and short frame image in Raw format from the DDR, align the long frame image to the short frame image to obtain the third alignment parameter. Obtain the next time-lapse long frame image and short frame image in Raw format from the DDR, align the long frame image to the short frame image to obtain the fourth alignment parameter.

[0113] S603: Obtain the current time-of-flight short frame image in Raw format from the DDR, process the current time-of-flight short frame image in Raw format through the Raw domain processing module to obtain the current time-of-flight short frame image in RGB format. Obtain the next time-of-flight short frame image in Raw format from the DDR, process the next time-of-flight short frame image in Raw format through the Raw domain processing module to obtain the next time-of-flight short frame image in RGB format.

[0114] S604: Obtain the current-time Raw format long-frame image from the DDR, and use the current-time Raw format long-frame image and the third alignment parameter as input to the RGB domain alignment module to obtain a new long-frame image in Raw format for the current time. Obtain the next-time Raw format long-frame image from the DDR, and use the next-time Raw format long-frame image and the fourth alignment parameter as input to the RGB domain alignment module to obtain a new long-frame image in Raw format for the next time.

[0115] Specifically, the current long frame image in Raw format is converted into a long frame image in RGB format, and then, based on the third alignment parameter, the long frame image in RGB format is aligned with the short frame image to obtain a new long frame image in Raw format.

[0116] S605: The new long frame image and short frame image in RGB format at the current moment are processed by the RGB domain HDR module to obtain the current frame image in RGB format at the current moment. The new long frame image and short frame image in RGB format at the next moment are processed by the RGB domain HDR module to obtain the next frame image in RGB format at the next moment.

[0117] S606: Align the next frame image in RGB format with the current frame image in RGB format to obtain a second alignment parameter. Convert the current frame image from RGB format to YUV format and the next frame image from RGB format to YUV format. Based on the second alignment parameter, align the next frame image in YUV format with the current frame image in YUV format to obtain a new next frame image in YUV format. Through the YUV domain HDR module, use the new next frame image in YUV format to perform denoising processing on the current frame image in YUV format to obtain the first denoised image of the current frame in YUV format and store it in DDR.

[0118] Here, the YUV domain HDR module can be used as a multi-frame noise reduction (MFNR) module based on a finite impulse response filter (FIR) for noise reduction processing.

[0119] S607: Obtain the reference frame image of the previous frame in YUV format from DDR, process the reference frame image of the previous frame in YUV format through the YUV domain alignment module, and obtain a new reference frame image of the previous frame that is aligned with the second denoised image of the current frame.

[0120] S608: The first denoised image of the current frame in YUV format and the new reference frame image of the previous frame in YUV format are processed by the first MCTF module to obtain the reference frame image of the current frame in YUV format, and stored in DDR.

[0121] The first MCTF is a YUV domain time-domain denoising algorithm that uses the Infinite Impulse Response (IIR) method.

[0122] S609: The reference frame image of the current frame in YUV format is processed by the YUV domain module to obtain the processed reference frame image, and stored in DDR for use in video recording and preview.

[0123] The YUV domain module includes at least the following processing operations: image stabilization, distortion correction, noise reduction, enhancement, and scaling.

[0124] This application also illustrates an image denoising method. Figure 7 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 7 ,like Figure 7 As shown, image denoising methods may include:

[0125] Figure 7 S601 to S605 and Figure 6 The content of S601 to S605 is the same, so it will not be repeated here.

[0126] S610: Align the next frame image in RGB format with the current frame image in RGB format to obtain a second alignment parameter; convert the current frame image from RGB format to YUV format and the next frame image from RGB format to YUV format; based on the second alignment parameter, align the next frame image in YUV format with the current frame image in YUV format to obtain a new next frame image in YUV format; through the second MCTF module, use the new next frame image in YUV format to perform denoising processing on the current frame image in YUV format to obtain a first denoised image of the current frame in YUV format, and store it in DDR.

[0127] Here, the second MCTF module in the YUV domain can be used as a multi-frame noise reduction (MFNR) module based on a finite impulse response filter (FIR) for noise reduction processing.

[0128] Figure 7 S607 to S609 and Figure 6 The content of S607 to S609 is the same, so it will not be repeated here.

[0129] This application also illustrates an image denoising method. Figure 8 This is a flowchart illustrating the image denoising method in the embodiments of this application. Figure 8 ,like Figure 8 As shown, image denoising methods may include:

[0130] Figure 8 S601 to S605 and Figure 6 The content of S601 to S605 is the same, so it will not be repeated here.

[0131] In this embodiment, S605 also stores the current frame image and the next frame image in RGB format into DDR.

[0132] The third MCTF includes the second MCTF and the first MCTF.

[0133] S611: Align the RGB format of the next frame image with the RGB format of the current frame image to obtain a second alignment parameter. Convert the current frame image from RGB format to YUV format, and convert the next frame image from RGB format to YUV format. Based on the second alignment parameter, align the YUV format of the next frame image with the YUV format of the current frame image to obtain a new YUV format next frame image. Through the second MCTF module, use the new YUV format next frame image to denoise the YUV format current frame image to obtain a first denoised image of the current frame in YUV format. Next, process the first denoised image of the current frame in YUV format and the new reference frame image of the previous frame through the first MCTF module to obtain a reference frame image of the current frame in YUV format, and store it in DDR.

[0134] Figure 8 S607 and S609 Figure 6 The content of S607 and S609 is the same, so it will not be repeated here.

[0135] To implement the method of the embodiments of this application, based on the same inventive concept, an image denoising device is also provided in the embodiments of this application. Figure 9This is a schematic diagram of the structure of the image denoising device in the embodiments of this application, as shown below. Figure 9 As shown, the image denoising device 90 includes:

[0136] The acquisition unit 901 is used to acquire the current frame image and the next frame image; wherein the exposure time of the current frame image and the next frame image is equal;

[0137] The processing unit 902 is configured to denoise the current frame image using the next frame image to obtain a first denoised image of the current frame; and to perform motion estimation and compensation on the first denoised image of the current frame and the obtained reference frame image of the previous frame to obtain a reference frame image of the current frame; and to display the reference frame image of the current frame.

[0138] In this embodiment, the current frame image is first denoised using the next frame image to obtain the first denoised image of the current frame. Then, the first denoised image of the current frame is denoised again using the reference frame image of the previous frame, so that the noise of the final reference frame image of the current frame is greatly reduced, that is, the denoising effect is improved and the clarity of the reference frame image of the current frame is improved.

[0139] In some embodiments, the processing unit 902 is further configured to align the subsequent frame image in Raw format to the current frame image in Raw format to obtain a first alignment parameter; convert the current frame image from Raw format to RGB format, and convert the subsequent frame image from Raw format to RGB format; based on the first alignment parameter, align the subsequent frame image in RGB format to the current frame image in RGB format to obtain a new subsequent frame image in RGB format; and use the new subsequent frame image in RGB format to perform denoising processing on the current frame image in RGB format to obtain a first denoised image of the current frame in RGB format.

[0140] In some embodiments, the processing unit 902 is further configured to convert the first denoised image of the current frame from RGB format to YUV format; align the reference frame image of the previous frame in YUV format to the first denoised image of the current frame in YUV format to obtain a new reference frame image of the previous frame in YUV format; and perform motion estimation and compensation on the first denoised image of the current frame in YUV format and the new reference frame image of the previous frame in YUV format to obtain a reference frame image of the current frame in YUV format.

[0141] In some embodiments, the processing unit 902 is further configured to align the next frame image in RGB format to the current frame image in RGB format to obtain a second alignment parameter; convert the current frame image from RGB format to YUV format, and convert the next frame image from RGB format to YUV format; based on the second alignment parameter, align the next frame image in YUV format to the current frame image in YUV format to obtain a new next frame image in YUV format; and use the new next frame image in YUV format to perform denoising processing on the current frame image in YUV format to obtain a first denoised image of the current frame in YUV format.

[0142] In some embodiments, the acquisition unit 901 is further configured to acquire a long frame image and a short frame image in Raw format at the current moment, and to acquire a long frame image and a short frame image in Raw format at the next moment; the processing unit 902 is further configured to perform high dynamic range fusion processing on the long frame image and the short frame image in Raw format at the current moment in the RGB domain to obtain a current frame image in RGB format; and to perform high dynamic range fusion processing on the long frame image and the short frame image in Raw format at the next moment in the RGB domain to obtain a next frame image in RGB format.

[0143] In some embodiments, the processing unit 902 is further configured to convert the long frame image from Raw format to RGB format, and to convert the short frame image from Raw format to RGB format; align the RGB format long frame image to the short frame image to obtain a new RGB format long frame image; and use the new RGB format long frame image to perform noise reduction processing on the RGB format short frame image to obtain the current frame image in RGB format.

[0144] In some embodiments, the processing unit 902 is further configured to store the reference frame image of the current frame into a preset storage space.

[0145] This application also provides another electronic device. Figure 10 This is a schematic diagram of the structure of the electronic devices in the embodiments of this application, such as... Figure 10 As shown, the electronic device 100 includes: a processor 1001 and a memory 1002 configured to store computer programs capable of running on the processor;

[0146] When the processor 1001 is configured to run a computer program, it executes the method steps described in the foregoing embodiments.

[0147] Of course, in practical applications, such as Figure 10As shown, the various components in the electronic device 100 are coupled together via a bus system 1003. It is understood that the bus system 1003 is used to enable communication between these components. In addition to a data bus, the bus system 1003 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 10 The general labeled all buses as Bus System 1003.

[0148] In practical applications, the aforementioned processor can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the functions of the aforementioned processor can also be other types, and the embodiments of this application do not specifically limit this.

[0149] The aforementioned memory can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

[0150] In an exemplary embodiment, this application also provides a computer-readable storage medium for storing a computer program.

[0151] Optionally, the computer-readable storage medium can be applied to any of the methods in the embodiments of this application, and the computer program causes the computer to execute the corresponding processes implemented by the processor in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0152] For example, embodiments of this application also provide a computer program product, including a computer program that can be executed by a processor of an electronic device to perform the steps of any of the foregoing methods.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0154] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, in the various embodiments of the present invention, all functional units can be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units. Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0157] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0158] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0159] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An image denoising method, characterized in that, The method includes: Acquire the current frame image and the next frame image; wherein the exposure duration of the current frame image and the next frame image is equal; The current frame image is denoised using the next frame image to obtain the first denoised image of the current frame; Motion estimation and compensation are performed on the first denoised image of the current frame and the obtained reference frame image of the previous frame to obtain the reference frame image of the current frame; wherein, the reference frame image of the previous frame is a frame image that includes the image information of the previous frame and all previous frames, as well as the image information of the current frame. The image is displayed based on a reference frame image of the current frame.

2. The method according to claim 1, characterized in that, The step of denoising the current frame image using the subsequent frame image to obtain the first denoised image of the current frame includes: Align the next frame of the Raw format image with the current frame of the Raw format image to obtain the first alignment parameter; Convert the current frame image from Raw format to RGB format, and convert the next frame image from Raw format to RGB format; Based on the first alignment parameter, the next frame image in RGB format is aligned with the current frame image in RGB format to obtain a new next frame image in RGB format. The RGB format of the current frame image is denoised using the new next frame image in RGB format to obtain the first denoised image of the current frame in RGB format.

3. The method according to claim 2, characterized in that, The step of performing motion estimation and compensation on the first denoised image of the current frame and the obtained reference frame image of the previous frame to obtain the reference frame image of the current frame includes: Convert the first denoised image of the current frame from RGB format to YUV format; Align the reference frame image of the previous frame in YUV format with the first denoised image of the current frame in YUV format to obtain a new reference frame image of the previous frame in YUV format. Motion estimation and compensation are performed on the first denoised image of the current frame in YUV format and the new reference frame image of the previous frame in YUV format to obtain the reference frame image of the current frame in YUV format.

4. The method according to claim 1, characterized in that, The step of denoising the current frame image using the subsequent frame image to obtain the first denoised image of the current frame includes: Align the next frame image in RGB format with the current frame image in RGB format to obtain the second alignment parameter; Convert the current frame image from RGB format to YUV format, and convert the next frame image from RGB format to YUV format; Based on the second alignment parameter, the next frame image in YUV format is aligned with the current frame image in YUV format to obtain a new next frame image in YUV format. The current frame image in YUV format is denoised using the new next frame image in YUV format to obtain the first denoised image of the current frame in YUV format.

5. The method according to claim 4, characterized in that, Before acquiring the current frame image and the next frame image, the method further includes: Get the current long-frame and short-frame images in Raw format, and get the next long-frame and short-frame images in Raw format. In the RGB domain, the long frame image and short frame image in Raw format at the current time are subjected to high dynamic range fusion processing to obtain the current frame image in RGB format; In the RGB domain, the long frame image and the short frame image in Raw format at the next moment are subjected to high dynamic range fusion processing to obtain the next frame image in RGB format.

6. The method according to claim 5, characterized in that, The step of performing high dynamic range fusion processing on the current time's Raw format long-frame image and short-frame image in the RGB domain to obtain the current frame image in RGB format includes: Align the long frame image in Raw format to the short frame image in Raw format to obtain the third alignment parameter; Convert the long frame image from Raw format to RGB format, and convert the short frame image from Raw format to RGB format; Based on the third alignment parameter, the long frame image in RGB format is aligned to the short frame image in RGB format to obtain a new long frame image in RGB format. The RGB format short frame image is denoised using a new long frame image in RGB format to obtain the current frame image in RGB format.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Store the reference frame image of the current frame to a preset storage space.

8. An electronic device, characterized in that, The electronic device includes: a processor and a memory configured to store computer programs capable of running on the processor. Wherein, when the processor is configured to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images