Gradient-guided multi-frame image denoising method, device, equipment and medium

Abstract

The application provides a gradient-guided multi-frame image denoising method, device, equipment and medium, comprising: acquiring multiple images under the same scene and determining a reference image and N other images; aligning the N other images with the reference image; calculating gradient information of the reference image, creating a gradient guide image according to the gradient information to divide the reference image into texture regions; calculating the pixel value of each target pixel point on the reference image to obtain a denoised image, when calculating the pixel value of a current target pixel point, according to the texture region type where the current target pixel point is located, the current target pixel point, the current surrounding pixel point, the pixel points at the corresponding positions of the current target pixel point and the current surrounding pixel point on the N aligned images are determined as current to-be-fused pixel points, and the pixel value of each current to-be-fused pixel point is obtained according to the pixel value and position information of each current to-be-fused pixel point. The application improves the image denoising effect, retains image details, and reduces the calculation complexity.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a gradient-guided multi-frame image denoising method, apparatus, device, and medium. Background Technology

[0002] With the advancement of computer and multimedia technologies, people's demand for photography is increasing. However, due to the characteristics of digital devices, captured images are often affected by noise, leading to a decline in image quality. While existing single-frame image denoising algorithms are computationally simple, they often lose image details. To address this issue, multi-frame image denoising algorithms have emerged. These algorithms utilize multiple images captured at different times, employing a weighted fusion method for denoising, thereby avoiding the destruction of image details. Typically, the multi-frame image denoising process involves: ① Frame selection to find the clearest reference frame; ② Image alignment; ③ Weight calculation; ④ Image fusion.

[0003] Multi-frame image fusion in the temporal domain can preserve image details to the greatest extent. However, for noisy scenes, temporal denoising alone is insufficient to achieve satisfactory results. A combination of temporal and spatial domain denoising is often used, but commonly used spatial denoising methods often introduce problems, easily affecting image details. While more complex spatial denoising algorithms can better preserve image details, the increased computational complexity is unacceptable for the real-time requirements of digital device photography.

[0004] Therefore, it is necessary to propose a gradient-guided multi-frame image denoising method, apparatus, device, and medium to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a gradient-guided multi-frame image denoising method, apparatus, device, and medium to improve the problem that commonly used temporal-domain combined spatial denoising algorithms are difficult to preserve image details, while complex temporal-domain combined spatial denoising algorithms have high computational complexity.

[0006] In a first aspect, the present invention provides a gradient-guided multi-frame image denoising method, the method comprising:

[0007] Acquire multiple frames of images captured consecutively for the same shooting scene.

[0008] Select one reference image from the multiple images and determine N other images, where N is a positive integer.

[0009] Based on the reference image, the other N frames are aligned to obtain N aligned images.

[0010] The gradient information of the reference image is calculated using an edge detection operator, wherein the gradient information includes vertical texture confidence and horizontal texture confidence.

[0011] A gradient guidance map is created based on the gradient information to divide the reference image into texture regions.

[0012] The denoised pixel value of each target pixel in the reference image is calculated to obtain the denoised image.

[0013] In one possible embodiment, when calculating the denoised pixel value of the current target pixel, the type of the texture region where the current target pixel is located is determined. Based on the type of the texture region, the current target pixel, the currently surrounding pixels within a set range around the current target pixel, and the pixels at corresponding positions of the current target pixel and the currently surrounding pixels on the aligned image of N frames are determined as the current pixels to be fused. Based on the pixel value and position information of each current pixel to be fused, the fusion weight of each current pixel to be fused is calculated. Based on the fusion weight and pixel value of each current pixel to be fused, the denoised pixel value of the current target pixel is calculated.

[0014] In one possible embodiment, the texture region type includes a vertical texture region, a quasi-vertical texture region, a flat texture region, a quasi-horizontal texture region, and a horizontal texture region.

[0015] Determining the type of the texture region where the current target pixel is located includes:

[0016] The difference between the absolute value of the horizontal texture confidence of the current target pixel and the absolute value of the vertical texture confidence of the current target pixel is calculated.

[0017] If the difference is less than the first threshold, the type of the texture region where the current target pixel is located is determined to be the vertical texture region.

[0018] If the difference is greater than or equal to the first threshold and less than the second threshold, the type of the texture region where the current target pixel is located is determined to be the quasi-vertical texture region.

[0019] If the difference is greater than or equal to the second threshold and less than the third threshold, the type of the texture region where the current target pixel is located is determined to be the flat texture region.

[0020] If the difference is greater than or equal to the third threshold and less than the fourth threshold, the type of the texture region where the current target pixel is located is determined to be the quasi-horizontal texture region.

[0021] If the difference is greater than or equal to the fourth threshold, the type of the texture region where the current target pixel is located is determined to be the horizontal texture region.

[0022] In one possible embodiment, the fusion weight of each current pixel to be fused is calculated based on its pixel value and position information, including:

[0023] Calculate the pixel weight of each pixel to be merged based on its pixel value.

[0024] Based on the position information of each current pixel to be merged, calculate the distance weight of each current pixel to be merged.

[0025] The fusion weight of each pixel to be fused is calculated based on its pixel weight and distance weight.

[0026] In one possible embodiment, the denoised pixel value of the current target pixel is calculated based on the fusion weight and pixel value of each of the current pixels to be fused, including:

[0027] The fusion weights of each of the current pixels to be fused are accumulated to obtain the weight sum of the current target pixels.

[0028] Calculate the product of the fusion weight and the corresponding pixel value for each current pixel to be fused, and sum the product values ​​to obtain the fusion sum of the current target pixels.

[0029] Calculate the ratio of the fused sum of the current target pixels to the weighted sum to obtain the denoised pixel value of the current target pixel.

[0030] In one possible embodiment, selecting a reference image from the multiple frames and determining N other images includes:

[0031] The gradient information of the multi-frame images is calculated to evaluate the sharpness of the multi-frame images. The image with the highest sharpness is selected from the multi-frame images and determined as the reference image. Images that do not meet the sharpness requirements are removed from the multi-frame images, and the remaining N frames are determined as other images.

[0032] In one possible embodiment, based on the reference image, N frames of the other images are aligned to obtain N aligned images, including:

[0033] The reference image and the other N frames are downsampled into multi-layer Gaussian pyramid images, wherein the multi-layer Gaussian pyramid images represent multiple image layers of different resolutions, and each image layer includes multiple image blocks.

[0034] Each frame of other images is aligned with the reference image layer by layer from the highest image layer to the lowest image layer to obtain N frames of aligned images.

[0035] Secondly, the present invention also provides a gradient-guided multi-frame image denoising apparatus, which includes modules / units for performing any of the possible design methods described in the first aspect. These modules / units can be implemented in hardware or by hardware executing corresponding software.

[0036] Thirdly, embodiments of the present invention provide an electronic device, including a processor and a memory. The memory stores one or more computer programs; when the one or more computer programs stored in the memory are executed by the processor, the electronic device is able to implement any of the possible design methods described in the first aspect.

[0037] Fourthly, this invention also provides a computer-readable storage medium comprising a computer program that, when run on an electronic device, causes the electronic device to perform any of the possible designs described in the first aspect.

[0038] Fifthly, embodiments of the present invention also provide a method comprising a computer program product, which, when the computer program product is run on an electronic device, causes the electronic device to perform any possible design of any of the above aspects.

[0039] The beneficial effects of this invention are as follows: Guided by gradient information, the reference image is divided into texture regions. Spatial fusion is performed using pixels with different spatial distributions for target pixels in different texture regions of the reference image, while temporal fusion is also performed with the corresponding pixels of the target pixels in their frame image. When calculating the fusion weights of the target pixels, pixel value weights and distance weights are considered, significantly improving the denoising effect of the reference image while preserving image details as much as possible. Simultaneously, the computational complexity of the fusion is reduced to meet the real-time requirements of digital device photography. Attached Figure Description

[0040] Figure 1 This is a schematic flowchart of the gradient-guided multi-frame image denoising method of the present invention.

[0041] Figure 2 This is a flowchart illustrating the implementation of the gradient-guided multi-frame image denoising method of the present invention.

[0042] Figure 3 This is a schematic diagram of gradient-guided pixel fusion in the gradient-guided multi-frame image denoising method of the present invention.

[0043] Figure 4 This is a schematic diagram of the gradient-guided multi-frame image denoising device of the present invention.

[0044] Figure 5 This is a schematic diagram of the structure of the electronic device of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] The technical terms involved in this invention will be explained below.

[0047] 1. YUV: YUV is a color encoding mode, where Y represents luminance, which is the gray value, and UV represents chrominance and chroma, respectively.

[0048] 2. Adjusting parameters: Adapt to different scenarios by setting parameters of different intensities.

[0049] 3. Vertical texture confidence G x It is a metric for measuring vertical texture in an image. Vertical texture confidence is typically calculated by evaluating the vertical texture intensity of each pixel in the image.

[0050] 4. Horizontal texture confidence G y It is a metric for measuring lateral texture in an image. Lateral texture confidence is typically calculated by evaluating the lateral texture intensity of each pixel in the image.

[0051] With the development of technology, people have increasingly higher requirements for photography technology. Because imaging elements are affected by thermal noise, images directly captured by digital devices (such as cameras, DSLRs, and tablets) exhibit significant noise, impacting the visual experience. Commonly used temporal-spatial denoising algorithms struggle to preserve image details, while complex temporal-spatial denoising algorithms have high computational complexity and cannot meet the real-time requirements of digital device photography.

[0052] To address the aforementioned technical problems, embodiments of the present invention provide a gradient-guided multi-frame image denoising method, applicable to digital devices with camera functions, such as mobile phones, digital SLR cameras, and tablet computers. It can also be embedded into an image signal processor (ISP) chip to improve processing speed. See [link to relevant documentation]. Figure 1 and Figure 2 The method includes:

[0053] S101: Acquire multiple frames of images captured consecutively for the same shooting scene.

[0054] S102: Select a reference image from multiple frames and determine N other images, where N is a positive integer.

[0055] S103: Based on the reference image, align the other N frames to obtain the N-frame aligned image.

[0056] S104: Calculate the gradient information of the reference image using an edge detection operator, whereby the gradient information includes the vertical texture confidence G. x and horizontal texture confidence G y Preferably, edge detection operators can be the Sobel operator, Roberts operator, Prewitt operator, Laplace operator, Canny operator, or gradient detection operator, etc.

[0057] S105: Create a gradient guidance map based on gradient information to divide the reference image into texture regions.

[0058] S106: Calculate the denoised pixel value of each target pixel in the reference image to obtain the denoised image.

[0059] In a preferred embodiment, when calculating the denoised pixel value of the current target pixel, the type of the texture region where the current target pixel is located is determined. Based on the type of the texture region, the current target pixel, the surrounding pixels within a defined range around the current target pixel, and the pixels at the corresponding positions of the current target pixel and its surrounding pixels on the N-frame aligned image are identified as the current pixels to be fused. Based on the pixel value and position information of each current pixel to be fused, a fusion weight is calculated for each current pixel to be fused. Based on the fusion weight and pixel value of each current pixel to be fused, the denoised pixel value of the current target pixel is calculated. Preferably, the defined ranges are different for different types of texture regions.

[0060] In this embodiment, different ranges are set for target pixels in different texture regions to achieve the best denoising effect. For example, pixels in a vertical texture region are densely arranged in the vertical direction, so a suitable range can be determined in the vertical direction. Moreover, by selecting pixels with a suitable spatial distribution range to be fused, the number of pixels to be fused can be controlled, reducing the amount of fusion calculation and computational complexity to meet the real-time requirements of digital device photography. Temporal denoising is performed on the current target pixel and the corresponding pixels in the image after N-frame alignment. At the same time, the surrounding pixels and the corresponding pixels in the image after N-frame alignment are also included in the spatial-temporal denoising, which helps to improve the image denoising effect. This invention proposes a gradient-guided multi-frame denoising algorithm. Under the guidance of gradient information, different fusion methods are used for different texture regions, which can significantly suppress noise in the image, while making the image have smooth edges, less prone to edge noise, and better preservation of image details.

[0061] In one specific embodiment, selecting a reference image from multiple frames and determining N other images includes: calculating gradient information of the multiple frames to evaluate the sharpness of the multiple frames, selecting the image with the highest sharpness from the multiple frames as the reference image, removing images from the multiple frames that do not meet the sharpness requirements, and determining the remaining N frames as other images.

[0062] In one specific embodiment, based on a reference image, N other images are aligned to obtain N aligned images. This includes: downsampling the reference image and the N other images into multi-layer Gaussian pyramid images, where each multi-layer Gaussian pyramid image represents multiple image layers of different resolutions, and each image layer includes multiple image blocks. Each other image is then aligned with the reference image layer by layer, from the highest image layer to the lowest image layer, to obtain the N aligned images. In other words, the reference image and the N other images are downsampled into 4-layer Gaussian pyramid images. When aligning the current other images, starting from the highest layer, the image blocks of each layer of the current other image are aligned with the corresponding image blocks of the reference image. The position of the most matching image block in each layer of the current other image is found, and the matching image block position information is passed down layer by layer to obtain the position information of all image blocks in the current other image aligned with the reference frame, thus completing the alignment of the current other image with the reference image.

[0063] In one specific embodiment, after obtaining N aligned images, the method further includes: filtering the reference image to obtain a filtered image; preferably, Gaussian filtering is used. Calculating gradient information of the reference image using an edge detection operator includes: calculating gradient information of the filtered image using an edge detection operator. Creating a gradient guidance map based on the gradient information to segment the texture region of the reference image includes: creating a gradient guidance map based on the gradient information to segment the texture region of the filtered image. Calculating the denoised pixel value of each target pixel in the reference image to obtain a denoised image includes: calculating the denoised pixel value of each target pixel in the filtered image to obtain a denoised image.

[0064] Preferably, the gradient information of the filtered image is calculated using an edge detection operator, including: passing the filtered image through a Sobel convolution factor to obtain the gradient information of the filtered image. The Sobel convolution factor uses two 3x3 convolution kernels (kernal_x and kernel_y).

[0065] kernal x ={-1,0,1,-2,0,2,-1,0,1};kernal y ={1,2,1,0,0,0,-1,-2,-1}.

[0066] In this embodiment, a 3x3 convolution kernel is used, which can greatly reduce the amount of computation to meet the real-time requirements of digital device photography.

[0067] In a preferred embodiment, the texture region types include vertical texture regions, quasi-vertical texture regions, flat texture regions, quasi-horizontal texture regions, and horizontal texture regions. Determining the type of texture region where the current target pixel resides includes: calculating the horizontal texture confidence G of the current target pixel. y The absolute value of the vertical texture confidence G of the current target pixel x The difference in absolute values ​​of the values ​​is used to determine the texture region type. If the difference is less than a first threshold, the texture region containing the current target pixel is determined to be a vertical texture region. If the difference is greater than or equal to the first threshold and less than a second threshold, the texture region containing the current target pixel is determined to be a quasi-vertical texture region. If the difference is greater than or equal to the second threshold and less than a third threshold, the texture region containing the current target pixel is determined to be a flat texture region. If the difference is greater than or equal to the third threshold and less than a fourth threshold, the texture region containing the current target pixel is determined to be a quasi-horizontal texture region. If the difference is greater than or equal to the fourth threshold, the texture region containing the current target pixel is determined to be a horizontal texture region. Preferably, the first threshold is -128, the second threshold is -25, the third threshold is 25, and the fourth threshold is 128.

[0068] For details, see Figure 3 Based on the gradient guidance map, the reference image is divided into 5 types of texture regions:

[0069]

[0070] Among them, grad guide The gradient guide map is represented by 1, which represents the vertical texture region, 2 represents the quasi-vertical texture region, 3 represents the flat texture region, 4 represents the quasi-horizontal texture region, and 5 represents the horizontal texture region.

[0071] In this embodiment, based on the vertical texture confidence C of the target pixel... x and horizontal texture confidence C y It can determine the type of texture region where the target pixel is located, and different fusion methods are used for different texture regions to minimize noise and preserve image details, thereby improving the denoising effect.

[0072] In a preferred embodiment, the fusion weight of each current pixel to be fused is calculated based on its pixel value and position information. This includes: calculating the pixel weight of each current pixel to be fused based on its pixel value; calculating the distance weight of each current pixel to be fused based on its position information; and calculating the fusion weight of each current pixel to be fused based on its pixel weight and distance weight. In this embodiment, when calculating the fusion weight of the target pixel, both pixel value weight and distance weight are considered. The pixel value weight measures the contribution of each current pixel to be fused's pixel value, while the distance weight considers the influence of the position of each current pixel to be fused in the spatial domain on the fusion process.

[0073] In one specific embodiment, the denoised pixel value of the current target pixel is calculated based on the fusion weight and pixel value of each current pixel to be fused. This includes: accumulating the fusion weights of each current pixel to be fused to obtain the weighted sum of the current target pixels; calculating the product of the fusion weight and the corresponding pixel value of each current pixel to be fused, and accumulating these products to obtain the fusion sum of the current target pixels; and calculating the ratio of the fusion sum of the current target pixels to the weighted sum to obtain the denoised pixel value of the current target pixel.

[0074] See Figure 3 The following section explains the fusion and denoising process of the reference image and the image after N frames are aligned. For ease of explanation of the formula, the reference image and the image after N frames are aligned are defined as the images to be fused.

[0075] Specifically, the denoised pixel value of the current target pixel located in the vertical texture region satisfies the following calculation formula:

[0076]

[0077]

[0078] Where merge_out1 represents the denoised pixel value of the current target pixel located in the vertical texture region, and p represents the pixel value of the current target pixel located in the vertical texture region before denoising. i,x,y This represents the pixel value at position (x, y) of the i-th image to be fused, where (x, y) represents the relative position with respect to the current target pixel located in the vertical texture region. This is an adjustable parameter for the pixel weights in the vertical texture region, and its value is not zero. i,x,y This represents the distance from the (x, y) position of the i-th image to be fused to the current target pixel located in the vertical texture region. The distance weight for the vertical texture region is an adjustable parameter and its value is not 0. sum This represents the sum of the weights of the current target pixel located in the vertical texture region. x r y These are used to limit the range of values ​​for x and y, respectively, which is the setting range of the current target pixel located in the vertical texture area.

[0079] Combination Figure 3 As can be seen, the first row represents the vertical texture regions located at the same position on the reference image and the image after alignment of N frames. In the reference image of the first row, black pixels represent the current target pixel located in the vertical texture region, and gray pixels represent the current surrounding pixels located in the vertical texture region. In the aligned image of the first row, black pixels represent the pixels at the corresponding positions of the current target pixel located in the vertical texture region on the aligned image, and gray pixels represent the pixels at the corresponding positions of the current surrounding pixels located in the vertical texture region on the aligned image.

[0080] Specifically, the denoised pixel value of the current target pixel located in the quasi-vertical texture region satisfies the following calculation formula:

[0081]

[0082]

[0083] Where merge_out2 represents the denoised pixel value of the current target pixel located in the quasi-vertical texture region, and p represents the pixel value of the current target pixel located in the quasi-vertical texture region before denoising. i,x,y Let (x, y) represent the pixel value at position (x, y) of the i-th image to be fused, where (x, y) represents the relative position with respect to the current target pixel located in the quasi-vertical texture region. The pixel weights for the quasi-vertical texture region are adjustable parameters and their values ​​are not zero. d i,x,y This represents the distance from the (x, y) position of the i-th image to be fused to the current target pixel located in the quasi-vertical texture region. The distance weight for the quasi-vertical texture region is an adjustable parameter and its value is not 0. sum This represents the sum of the weights of the current target pixel located in the quasi-vertical texture region. x r y These are used to limit the range of values ​​for x and y, respectively, which is the setting range of the current target pixel located in the quasi-vertical texture region.

[0084] Combination Figure 3 As can be seen, the second row represents the quasi-vertical texture regions located at the same position on the reference image and the image after alignment of N frames. In the reference image of the second row, the black pixels represent the current target pixel located in the quasi-vertical texture region, and the gray pixels represent the current surrounding pixels located in the quasi-vertical texture region. In the aligned image of the second row, the black pixels represent the pixels at the corresponding positions of the current target pixel located in the quasi-vertical texture region on the aligned image, and the gray pixels represent the pixels at the corresponding positions of the current surrounding pixels located in the quasi-vertical texture region on the aligned image.

[0085] Specifically, the denoised pixel value of the current target pixel located in a flat texture region satisfies the following calculation formula:

[0086]

[0087]

[0088] Where merge_out3 represents the denoised pixel value of the current target pixel located in the flat texture region, and p represents the pixel value of the current target pixel located in the flat texture region before denoising. i,x,y This represents the pixel value at position (x, y) in the i-th image to be fused, where (x, y) represents the relative position with respect to the current target pixel located in a flat texture region. The pixel weights for flat texture regions are adjustable parameters and their values ​​are not zero. d i,x,y This represents the distance from the (x, y) position of the i-th image to be fused to the current target pixel located in the flat texture region. The distance weight for flat texture regions is an adjustable parameter and its value is not 0. sum This represents the sum of the weights of the current target pixel located in a flat texture region. x r y These are used to limit the range of values ​​for x and y, that is, the range of the current target pixel located in the flat texture area.

[0089] Combination Figure 3 As can be seen, the third row represents the flat texture region located at the same position on the reference image and the image after alignment of N frames. In the reference image of the third row, the black pixels represent the current target pixel located in the flat texture region, and the gray pixels represent the current surrounding pixels located in the flat texture region. In the aligned image of the third row, the black pixels represent the pixels at the corresponding positions of the current target pixel located in the flat texture region on the aligned image, and the gray pixels represent the pixels at the corresponding positions of the current surrounding pixels located in the flat texture region on the aligned image.

[0090] Specifically, the denoised pixel value of the current target pixel located in the quasi-horizontal texture region satisfies the following calculation formula:

[0091]

[0092]

[0093] Where merge_out4 represents the denoised pixel value of the current target pixel located in the quasi-horizontal texture region, and p represents the pixel value of the current target pixel located in the quasi-horizontal texture region before denoising. i,x,y Let (x, y) represent the pixel value at position (x, y) of the i-th image to be fused, where (x, y) represents the relative position with respect to the current target pixel located in the quasi-horizontal texture region. The pixel weights for the quasi-horizontal texture region are adjustable parameters and their values ​​are not zero. d i,x,y This represents the distance from the (x, y) position of the i-th image to be fused to the current target pixel located in the quasi-horizontal texture region. The distance weight for the quasi-horizontal texture region is an adjustable parameter and its value is not 0. sum This represents the sum of the weights of the current target pixel located in the quasi-horizontal texture region. x r y These are used to limit the range of values ​​for x and y, that is, the range of the current target pixel located in the quasi-horizontal texture area.

[0094] Combination Figure 3 As can be seen, the fourth row represents the quasi-horizontal texture region located at the same position on the reference image and the image after alignment of N frames. In the reference image of the fourth row, the black pixels represent the current target pixel located in the quasi-horizontal texture region, and the gray pixels represent the current surrounding pixels located in the quasi-horizontal texture region. In the aligned image of the fourth row, the black pixels represent the pixels at the corresponding positions of the current target pixel located in the quasi-horizontal texture region on the aligned image, and the gray pixels represent the pixels at the corresponding positions of the current surrounding pixels located in the quasi-horizontal texture region on the aligned image.

[0095] Specifically, the denoised pixel value of the current target pixel located in the horizontal texture region satisfies the following calculation formula:

[0096]

[0097]

[0098] Where merge_out5 represents the denoised pixel value of the current target pixel located in the horizontal texture region, and p represents the pixel value of the current target pixel located in the horizontal texture region before denoising. i,x,y This represents the pixel value at position (x, y) of the i-th image to be fused, where (x, y) represents the relative position with respect to the current target pixel located in the horizontal texture region. This is an adjustable parameter for the pixel weights of the horizontal texture region, and its value is not zero. i,x,y This represents the distance from the (x, y) position of the i-th image to be fused to the current target pixel located in the horizontal texture region. The distance weight for the horizontal texture region is an adjustable parameter and its value is not 0. sum This represents the sum of the weights of the current target pixels located in the horizontal texture region. x r y These are used to limit the range of values ​​for x and y, that is, the range of the current target pixel located in the horizontal texture area.

[0099] Combination Figure 3 As can be seen, the fifth row represents the horizontal texture region located at the same position on the reference image and the image after alignment of N frames. In the reference image of the fifth row, the black pixels represent the current target pixel located in the horizontal texture region, and the gray pixels represent the current surrounding pixels located in the horizontal texture region. In the image after alignment of the fifth row, the black pixels represent the pixels at the corresponding positions of the current target pixel located in the horizontal texture region on the aligned image, and the gray pixels represent the pixels at the corresponding positions of the current surrounding pixels located in the horizontal texture region on the aligned image.

[0100] According to the gradient guidance graph grad guide By dividing the reference image into different texture regions and applying different fusion methods to the target pixels in different texture regions, a denoised image can be obtained:

[0101]

[0102] Where merge_out represents the denoised image, grad guideThe gradient guide map is represented by 1, 2, 3, 4, and 5. 1 represents the vertical texture region, 2 represents the quasi-vertical texture region, 3 represents the flat texture region, 4 represents the quasi-horizontal texture region, and 5 represents the horizontal texture region. merge_out1 represents the denoised pixel value of the target pixel located in the vertical texture region, merge_out2 represents the denoised pixel value of the target pixel located in the quasi-vertical texture region, merge_out3 represents the denoised pixel value of the target pixel located in the flat texture region, merge_out4 represents the denoised pixel value of the target pixel located in the quasi-horizontal texture region, and merge_out5 represents the denoised pixel value of the target pixel located in the horizontal texture region.

[0103] The gradient-guided multi-frame image denoising method of this invention operates in the YUV domain of an image, achieving a certain denoising effect on all three channels (Y, U, and V). In a single frame, pixels located in different texture regions undergo spatial fusion due to their spatially different distributions. Simultaneously, in the temporal multi-frame image, for each pixel, pixels with the same distribution across different images undergo temporal fusion. Based on different texture region divisions, pixels located in different texture regions are weighted in both the temporal and spatial domains using pixels with different spatial distributions. Each pixel undergoes a weighted average to obtain a clean denoised image, improving the denoising effect while preserving image details. This also reduces computational complexity to meet the real-time requirements of digital device photography.

[0104] In addition, this invention also proposes a gradient-guided multi-frame image denoising device, which can be applied to digital devices with camera functionality and image signal processor chips. See also Figure 4 The device includes: an acquisition unit 401, used to acquire multiple frames of images continuously captured for the same shooting scene; a determination unit 402, used to select a reference image from the multiple frames and determine N other images, where N is a positive integer; an alignment unit 403, used to align the N other images according to the reference image to obtain N aligned images; an edge detection unit 404, used to calculate the gradient information of the reference image using an edge detection operator, where the gradient information includes vertical texture confidence and horizontal texture confidence; a region segmentation unit 405, used to create a gradient guidance map according to the gradient information to segment the reference image into texture regions; and a denoising unit 406, used to calculate the denoised pixel value of each target pixel in the reference image to obtain a denoised image. All relevant content of each step involved in the above method embodiment can be referred to in the functional description of the corresponding functional module, and will not be repeated here.

[0105] In a preferred embodiment, when calculating the denoised pixel value of the current target pixel, the denoising unit 406 determines the type of the texture region where the current target pixel is located. Based on the type of the texture region, the current target pixel, the surrounding pixels within a set range around the current target pixel, and the pixels at the corresponding positions of the current target pixel and the surrounding pixels on the image after N frames are aligned are determined as the current pixels to be fused. Based on the pixel value and position information of each current pixel to be fused, the fusion weight of each current pixel to be fused is calculated. Based on the fusion weight and pixel value of each current pixel to be fused, the denoised pixel value of the current target pixel is calculated.

[0106] In a preferred embodiment, the types of texture regions include vertical texture regions, quasi-vertical texture regions, flat texture regions, quasi-horizontal texture regions, and horizontal texture regions.

[0107] The denoising unit 406 determines the type of the texture region where the current target pixel is located, specifically for:

[0108] The difference between the absolute value of the horizontal texture confidence of the current target pixel and the absolute value of the vertical texture confidence of the current target pixel is calculated.

[0109] If the difference is less than the first threshold, the texture region where the current target pixel is located is determined to be a vertical texture region.

[0110] If the difference is greater than or equal to the first threshold and less than the second threshold, the texture region where the current target pixel is located is determined to be a quasi-vertical texture region.

[0111] If the difference is greater than or equal to the second threshold and less than the third threshold, the texture region where the current target pixel is located is determined to be a flat texture region.

[0112] If the difference is greater than or equal to the third threshold and less than the fourth threshold, the type of the texture region where the current target pixel is located is determined to be a quasi-horizontal texture region.

[0113] If the difference is greater than or equal to the fourth threshold, the texture region where the current target pixel is located is determined to be a horizontal texture region.

[0114] In a preferred embodiment, the denoising unit 406 calculates the fusion weight of each current pixel to be fused based on its pixel value and position information, specifically for:

[0115] Calculate the pixel weight of each pixel to be merged based on its pixel value.

[0116] Calculate the distance weight of each pixel to be merged based on its position information.

[0117] Calculate the fusion weight of each current pixel to be fused based on its pixel weight and distance weight.

[0118] In a preferred embodiment, the denoising unit 406 calculates the denoised pixel value of the current target pixel based on the fusion weight and pixel value of each current pixel to be fused, specifically for:

[0119] The fusion weights of each current pixel to be fused are summed to obtain the weight sum of the current target pixel.

[0120] Calculate the product of the fusion weight and the corresponding pixel value for each current pixel to be fused, and sum the products to obtain the fusion sum of the current target pixels.

[0121] Calculate the ratio of the fused sum of the current target pixels to the weighted sum to obtain the denoised pixel value of the current target pixel.

[0122] In a preferred embodiment, the determining unit 402 selects a reference image from multiple frames of images and determines N other frames, specifically for:

[0123] The gradient information of multiple frames of images is calculated to evaluate the sharpness of the multiple frames of images. The image with the highest sharpness is selected from the multiple frames of images and determined as the reference image. Images that do not meet the sharpness requirements are removed from the multiple frames of images, and the remaining N frames of images are determined as other images.

[0124] In a preferred embodiment, the alignment unit 403 aligns N other images according to the reference image to obtain N aligned images, specifically for:

[0125] The reference image and N other images are downsampled into multi-layer Gaussian pyramid images, where the multi-layer Gaussian pyramid image represents multiple image layers of different resolutions, and each image layer includes multiple image blocks.

[0126] Each frame of other images is aligned with the reference image layer by layer from the highest image layer to the lowest image layer to obtain N frames of aligned images.

[0127] In other embodiments of the present invention, an electronic device is disclosed, see [link to relevant documentation]. Figure 5The electronic device may include: one or more processors 501; a memory 502; a display 503; one or more application programs (not shown); and one or more computer programs 504. These devices may be connected via one or more communication buses 505. The one or more computer programs 504 are stored in the memory 502 and configured to be executed by the one or more processors 501. The one or more computer programs 504 include instructions that can be used to perform actions such as… Figure 1 and Figure 4 And the steps in the corresponding embodiments.

[0128] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0129] In the various embodiments of this invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0131] While embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations fall within the scope and spirit of the invention as set forth in the claims. Furthermore, the invention described herein may have other embodiments and can be implemented or carried out in various ways. Unless otherwise defined, the technical or scientific terms used herein should be understood in their ordinary sense by one of ordinary skill in the art to which this invention pertains. The terms "comprising" and similar expressions used herein mean that the element or object preceding the word encompasses the element or object listed following the word and its equivalents, but do not exclude other elements or objects.

Claims

1. A gradient-guided multi-frame image denoising method, characterized in that, The method includes: Acquire multiple frames of images captured consecutively for the same shooting scene; Select one reference image from the multiple frames of images, and determine N other images, where N is a positive integer; Based on the reference image, the other N frames are aligned to obtain N aligned images; The gradient information of the reference image is calculated using an edge detection operator, wherein the gradient information includes vertical texture confidence and horizontal texture confidence; A gradient guidance map is created based on the gradient information to divide the reference image into texture regions; The denoised pixel value of each target pixel in the reference image is calculated to obtain the denoised image. When calculating the denoised pixel value of the current target pixel, the type of the texture region where the current target pixel is located is determined. Based on the type of the texture region, the current target pixel, the surrounding pixels within a set range around the current target pixel, and the pixels at the corresponding positions of the current target pixel and the surrounding pixels in the aligned image of N frames are determined as the current pixels to be fused. Based on the pixel value and position information of each current pixel to be fused, the fusion weight of each current pixel to be fused is calculated. Based on the fusion weight and pixel value of each current pixel to be fused, the denoised pixel value of the current target pixel is calculated.

2. The method according to claim 1, characterized in that, The types of texture regions include vertical texture regions, quasi-vertical texture regions, flat texture regions, quasi-horizontal texture regions, and horizontal texture regions; Determining the type of the texture region where the current target pixel is located includes: The difference between the absolute value of the horizontal texture confidence of the current target pixel and the absolute value of the vertical texture confidence of the current target pixel is calculated. If the difference is less than the first threshold, the type of the texture region where the current target pixel is located is determined to be the vertical texture region. If the difference is greater than or equal to the first threshold and less than the second threshold, the type of the texture region where the current target pixel is located is determined to be the quasi-vertical texture region. If the difference is greater than or equal to the second threshold and less than the third threshold, the type of the texture region where the current target pixel is located is determined to be the flat texture region. If the difference is greater than or equal to the third threshold and less than the fourth threshold, the type of the texture region where the current target pixel is located is determined to be the quasi-horizontal texture region. If the difference is greater than or equal to the fourth threshold, the type of the texture region where the current target pixel is located is determined to be the horizontal texture region.

3. The method according to claim 1 or 2, characterized in that, Based on the pixel value and position information of each current pixel to be fused, calculate the fusion weight of each current pixel to be fused, including: Calculate the pixel weight of each pixel to be merged based on its pixel value; Calculate the distance weight of each pixel to be fused based on its position information; The fusion weight of each pixel to be fused is calculated based on its pixel weight and distance weight.

4. The method according to claim 3, characterized in that, Based on the fusion weight and pixel value of each current pixel to be fused, the denoised pixel value of the current target pixel is calculated, including: The fusion weights of each current pixel to be fused are summed to obtain the weight sum of the current target pixels; Calculate the product of the fusion weight and the corresponding pixel value for each of the current pixels to be fused, and sum the product values ​​to obtain the fusion sum of the current target pixels; Calculate the ratio of the fused sum of the current target pixels to the weighted sum to obtain the denoised pixel value of the current target pixel.

5. The method according to any one of claims 1-2, characterized in that, Selecting a reference image from the multiple frames and determining N other images includes: The gradient information of the multi-frame images is calculated to evaluate the sharpness of the multi-frame images. The image with the highest sharpness is selected from the multi-frame images and determined as the reference image. Images that do not meet the sharpness requirements are removed from the multi-frame images, and the remaining N frames are determined as other images.

6. The method according to any one of claims 1-2, characterized in that, Based on the reference image, N frames of the other images are aligned to obtain N aligned images, including: The reference image and the other N frames are downsampled into multi-layer Gaussian pyramid images, wherein the multi-layer Gaussian pyramid images represent multiple image layers of different resolutions, and each image layer includes multiple image blocks; Each frame of other images is aligned with the reference image layer by layer from the highest image layer to the lowest image layer to obtain N frames of aligned images.

7. A gradient-guided multi-frame image denoising device, characterized in that, The device includes: The acquisition unit is used to acquire multiple frames of images captured continuously for the same shooting scene. The determining unit is used to select a reference image from the multiple frames of images and determine N other images, where N is a positive integer; An alignment unit is used to align N frames of the other images according to the reference image to obtain N frames of aligned images. An edge detection unit is used to calculate gradient information of the reference image using an edge detection operator, wherein the gradient information includes vertical texture confidence and horizontal texture confidence; A region segmentation unit is used to create a gradient guidance map based on the gradient information to segment the reference image into texture regions. The denoising unit is used to calculate the denoised pixel value of each target pixel in the reference image to obtain a denoised image. When calculating the denoised pixel value of the current target pixel, the type of the texture region where the current target pixel is located is determined. Based on the type of the texture region, the current target pixel, the surrounding pixels within a set range around the current target pixel, and the pixels at the corresponding positions of the current target pixel and the surrounding pixels in the aligned image of N frames are determined as the current pixels to be fused. Based on the pixel value and position information of each current pixel to be fused, the fusion weight of each current pixel to be fused is calculated. Based on the fusion weight and pixel value of each current pixel to be fused, the denoised pixel value of the current target pixel is calculated.

8. An electronic device, characterized in that, include: Processor and memory, wherein the memory is used to store computer programs; The processor is configured to execute a computer program stored in the memory to cause the electronic device to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images