Multi-frame image denoising method and device, terminal, chip and chip module

Abstract

A multi-frame image denoising method and device, a terminal, a chip and a chip module, the method comprising: determining a first pixel block in a first image according to a current target pixel point in the first image; determining a second pixel block matched with the first pixel block in each second image; for each second image, determining a weight block of the second image according to a similarity between the second pixel block and the first pixel block; for each second image, updating a first weight accumulation map and a first weight pixel product accumulation map according to a position of the first pixel block in the first image and the weight block; if the current target pixel point is the last target pixel point, obtaining a second weight accumulation map, and obtaining a second weight pixel product accumulation map; and obtaining a denoised image according to the second weight pixel product accumulation map and the second weight accumulation map. Through the scheme provided in the application, the denoising performance of multi-frame images can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for denoising multi-frame images, a terminal, a chip, and a chip module. Background Technology

[0002] With the continuous development of science and technology, digital photography technology has brought great convenience to people's lives. However, in actual use, the images captured usually contain noise.

[0003] Digital photography works by converting light signals into electrical signals using a photosensitive element, thus creating an image. Photosensitive elements can be charge-coupled devices (CCDs), complementary metal-oxide-semiconductor (CMOS), etc. Because photosensitive elements are inevitably affected by thermal noise, the image directly output from the photosensitive element will be affected by noise.

[0004] In addition, the shooting environment also introduces noise. For example, in low-light scenes (such as nighttime photography), to increase the brightness of the captured image, the sensitivity of the image sensor (i.e., the ISO value) is usually increased. However, increasing the sensitivity of the image sensor makes the electrical signal converted from the light signal more susceptible to noise interference. In low-light scenes, the signal also needs to be amplified, and during this amplification process, noise is also amplified simultaneously. Therefore, noise reduction processing is a crucial step in improving image quality.

[0005] Denoising typically includes single-frame denoising and multi-frame denoising. Single-frame denoising algorithms have relatively low computational complexity, but they can easily affect image details during denoising, thus reducing image quality. Multi-frame denoising, on the other hand, processes multiple frames of images taken of the same scene to obtain a single processed image. Compared to single-frame denoising, multi-frame denoising preserves more scene details and suppresses noise in the resulting image.

[0006] Theoretically, image noise can be viewed as a random variable following a certain distribution. The variance of this random variable reflects the strength of the noise; the larger the variance, the stronger the noise, and the smaller the variance, the weaker the noise. By processing multiple frames of the same scene, it is theoretically possible to reduce the variance of this noise random variable without affecting the original image, that is, to reduce the noise in the image without affecting the details of the image itself. Therefore, multi-frame image denoising technology has become an important digital image denoising method.

[0007] In practical applications, the denoising performance of existing multi-frame denoising methods still needs to be improved. Summary of the Invention

[0008] The technical problem to be solved in this application is how to improve noise reduction performance.

[0009] To address the aforementioned technical problems, this application provides a multi-frame image denoising method. The method includes: the multi-frame image includes a first image and at least one second image; the method includes: determining a first pixel block in the first image based on a current target pixel point in the first image, wherein the current target pixel point depends on a previous target pixel point and a preset step size value, the current target pixel point is located at a preset position of the first pixel block, the size of the first pixel block is m×n, and the preset step size value step satisfies: step < m and / or step < n; determining a second pixel block matching the first pixel block in each frame of the second image; for each frame of the second image, determining a weight block of the second image based on the similarity between the second pixel block and the first pixel block, wherein the weight block... The weight values ​​at each position in the heavy block are positively correlated with the similarity. For each frame of the second image, based on the position of the first pixel block in the first image, the weight block is accumulated to the first weight accumulation map of the second image to update the first weight accumulation map. Additionally, the product of the weight block and the second pixel block is accumulated to the first weight pixel product accumulation map of the second image to update the first weight pixel product accumulation map. If the current target pixel is the last target pixel, the first weight accumulation maps of each frame of the second image are accumulated to obtain a second weight accumulation map. The first weight pixel product accumulation map of each frame of the second image is also accumulated to obtain a second weight pixel product accumulation map. Based on the second weight pixel product accumulation map and the second weight accumulation map, the denoised image is obtained.

[0010] Optionally, the preset step size is 1.

[0011] Optionally, the preset position is the center position.

[0012] Optionally, the second pixel block includes a second central pixel block and a second neighboring pixel block. Determining the second pixel block that matches the first pixel block in each frame of the second image includes: determining the pixel point corresponding to the current target pixel point in each frame of the second image based on the current target pixel point, and denoting it as the matching pixel point; determining the second central pixel block and the second neighboring pixel block based on the matching pixel point; wherein the matching pixel point is located at a preset position of the second central pixel block, and the matching pixel point is located at the peripheral position of the second neighboring pixel block, and the peripheral position refers to other positions besides the preset position.

[0013] Optionally, the size of the first pixel block is 3×3, the preset position is the 2nd row and 2nd column, and the surrounding position includes any one or more of the following: the 1st row and 2nd column, the 2nd row and 1st column, the 3rd row and 2nd column, and the 2nd row and 3rd column.

[0014] Optionally, both the first image and the at least one frame of the second image include Q M×N image blocks. The current target pixel is the pixel in the i-th row and j-th column of the q-th image block in the first image. Determining the second pixel block that matches the first pixel block in each frame of the second image includes: determining the image block with the highest similarity to the q-th image block among the Q image blocks in each frame of the second image, and denoting it as the matching image block of the q-th image block; determining the second pixel block in each frame of the second image based on the pixel in the i-th row and j-th column of the matching image block of the q-th image block; wherein the pixel in the i-th row and j-th column of the matching image block is located at a preset position of the second pixel block, 1≤q≤Q, 1≤i≤M, 1≤j≤N, and Q, M, and N are positive integers.

[0015] Optionally, the weight values ​​in the weight block are calculated and determined according to the following formula:

[0016]

[0017] Among them, W p is the weight value in the weight block, sigma is a preset parameter value, and diff is the difference between the second pixel block and the first pixel block.

[0018] Optionally, obtaining the denoised image based on the second weighted pixel multiplication accumulation map and the second weight accumulation map includes: accumulating the second weighted pixel multiplication accumulation map and the weighted pixel multiplication map of the first image to obtain a third weighted pixel multiplication accumulation map; accumulating the second weight accumulation map and the weight map of the first image to obtain a third weight accumulation map; and dividing the value at each position in the third weighted pixel multiplication accumulation map by the value at the corresponding position in the third weight accumulation map to obtain the pixel value at the corresponding position in the denoised image.

[0019] Optionally, before determining the first pixel block in the first image based on the current target pixel in the first image, the method further includes: removing images that do not meet a preset standard from the multiple frames of images to obtain a removed image, wherein the removed image includes the first image and the at least one frame of the second image.

[0020] To address the aforementioned technical features, this application also provides a multi-frame image denoising apparatus. The multi-frame image includes a first image and at least one second image. The apparatus includes: a first determining module, configured to determine a first pixel block in the first image based on a current target pixel in the first image, wherein the current target pixel depends on a previous target pixel and a preset step value, the current target pixel is located at a preset position of the first pixel block, the size of the first pixel block is m×n, and the preset step value step satisfies: step < m and / or step < n; a second determining module, configured to determine a second pixel block matching the first pixel block in each frame of the second image; and a third determining module, configured to determine a weight block of each frame of the second image based on the similarity between the second pixel block and the first pixel block, wherein the weight block... The weight values ​​at each position in the image are positively correlated with the similarity. A first calculation module is configured to, for each frame of the second image, accumulate the weight block to a first weight accumulation map of the second image based on the position of the first pixel block in the first image, to update the first weight accumulation map; and accumulate the product of the weight block and the second pixel block to a first weight pixel multiplication accumulation map of the second image, to update the first weight pixel multiplication accumulation map. A second calculation module is configured to, if the current target pixel is the last target pixel, accumulate the first weight accumulation maps of each frame of the second image to obtain a second weight accumulation map, and accumulate the first weight pixel multiplication accumulation maps of each frame of the second image to obtain a second weight pixel multiplication accumulation map. A third calculation module is configured to obtain the denoised image based on the second weight pixel multiplication accumulation map and the second weight accumulation map.

[0021] This application also provides a computer-readable storage medium storing a computer program thereon, which, when run by a processor, performs the steps of the multi-frame image denoising method described above.

[0022] This application embodiment also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it performs the steps of the multi-frame image denoising method described above.

[0023] This application also provides a chip used in the steps of the above-described multi-frame image denoising method.

[0024] This application also provides a chip module, which is used in the steps of the above-described multi-frame image denoising method.

[0025] Compared with the prior art, the technical solution of this application embodiment has the following beneficial effects:

[0026] In the scheme of this application embodiment, the current target pixel and the first pixel block containing the current target pixel are sequentially determined in the first image. Then, the second pixel block corresponding to the first pixel block is determined in the second image, and the weight block of the second pixel block is calculated based on the similarity between each second pixel block and the first pixel block. Further, based on the weight value of the second pixel block, weight accumulation and weighted pixel multiplication accumulation are performed to update the first weight accumulation map and the first weighted pixel multiplication accumulation map of the second image. Whenever the target pixel is updated, the first weight accumulation map and the first weighted pixel multiplication accumulation map of each second image are updated. Finally, the denoised image is obtained by accumulating the second weight accumulation map obtained from each first weight accumulation map and the second weighted pixel multiplication accumulation map obtained by accumulating the first weighted pixel multiplication accumulation maps.

[0027] When adopting the above scheme, firstly, compared with the existing scheme, the above scheme directly calculates the weight value based on the first pixel block and the second pixel block, instead of performing alignment processing before calculating the weight value. Therefore, it does not introduce alignment error, and the weight value can more accurately and realistically represent the pixel degree between the first image content and the second image content.

[0028] Secondly, in the embodiments of this application, the weight block is calculated based on the similarity between the first pixel block and the second pixel block, rather than based on the similarity of a single pixel. The weight block can more comprehensively consider the image information around the target pixel, so that the value of the weight block can reflect a wider range of image information, thereby helping to improve the denoising effect.

[0029] Thirdly, since the preset step size does not exceed the size of the first pixel block, at least some pixels in the first image can be used multiple times to determine the weight block, which is beneficial to recover more image details during the denoising process, thereby improving the denoising effect.

[0030] Furthermore, in this embodiment, the second pixel block includes a second center pixel block and a second neighboring pixel block, thereby allowing multiple second pixel blocks to be determined for each first pixel block. This approach further enhances the image information surrounding the target pixel, enabling the weighted blocks to represent greater similarity between image content, enriching image details, and improving denoising performance. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating a multi-frame denoising method in the prior art;

[0032] Figure 2 This is a flowchart illustrating a multi-frame image denoising method according to an embodiment of this application;

[0033] Figure 3 This is a partial flowchart of another multi-frame image denoising method in the embodiments of this application;

[0034] Figure 4 This is a schematic diagram of the structure of a multi-frame image denoising device according to an embodiment of this application. Detailed Implementation

[0035] As described in the background section, due to the inherent characteristics of photographic equipment and the influence of the shooting environment, the captured images are inevitably affected by noise, so it is necessary to denoise the obtained images. Compared with single-frame denoising algorithms, which easily blur the details of the image itself and reduce image quality, multi-frame denoising algorithms can theoretically reduce the variance of the random distribution of noise, thereby fundamentally reducing image noise without affecting the inherent content of the image.

[0036] Reference Figure 1 , Figure 1 This is a flowchart illustrating a multi-frame denoising method in the prior art. Existing multi-frame denoising methods typically include steps S11 to S13.

[0037] Step S11: Align the multiple frames of images.

[0038] Specifically, during the photography process, there are often unavoidable device shakes or localized movements within the scene, resulting in some offset between the captured multi-frame images. Therefore, the frames other than the reference frame in the multi-frame images can be aligned separately to make the aligned frames as close as possible to the reference frame, that is, to make the content of the aligned frames as consistent as possible with the reference frame.

[0039] Step S12: Calculate the weight of each frame of the image after alignment processing.

[0040] Specifically, for each frame of image after alignment, the similarity between each pixel in the frame and the pixel at the same position in the reference frame is calculated, and the weight value of the pixel is determined based on the similarity between the pixels, thereby obtaining the weight map corresponding to each frame of image.

[0041] Step S13: Perform fusion and denoising on the reference frame and other frames after alignment processing.

[0042] Specifically, the reference frame also has a corresponding weight map. The reference frame and other frames after alignment are weighted and calculated with their respective weight maps to obtain the denoised image.

[0043] The inventors of this application have discovered that the noise reduction performance of the prior art needs to be improved due to the following aspects:

[0044] (1) The accuracy of the weight values ​​needs to be improved. As mentioned above, the basic principle of multi-frame image denoising is to perform a weighted average of corresponding pixels in multiple frames to reduce the variance of the random variable that the noise follows, thereby achieving the purpose of denoising. Therefore, the calculation of weight values ​​is crucial to the denoising effect. However, the alignment process in the existing technology usually leads to alignment errors, so directly using the aligned image for weighted fusion will inevitably affect image details and the denoising effect.

[0045] For example, in global alignment schemes, images cannot be accurately aligned when moving objects are present in the scene, leading to alignment errors. In local alignment schemes, the reference frame is divided into several equal-sized, non-overlapping image blocks. Image block matching is used to find the most similar equal-sized image block in other frames to each image block in the reference frame, and this found image block is then "placed" in its corresponding position to obtain the aligned image. However, alignment errors also exist between image blocks, and these errors accumulate in the final aligned image, inevitably affecting subsequent noise reduction.

[0046] (2) When calculating weights, only the similarity between pixels at a single location is considered. In existing solutions, since the weight values ​​are determined based on the similarity between pixels, the weight values ​​at each location in the weight map can usually only represent the degree of difference between pixels at the same location. Determining weight values ​​based only on the similarity between pixels at a single location, without considering surrounding image information, makes it difficult to recover image details affected by noise during the denoising process, thus affecting denoising performance.

[0047] (3) Each pixel only participates in a single weight calculation and a single fusion denoising, resulting in limited overall denoising capability of the algorithm. In particular, it cannot achieve satisfactory denoising results in noisy scenes.

[0048] To address the aforementioned technical problems, this application provides a multi-frame image denoising method. The solution provided in this application has the following advantages: First, compared to existing solutions, the solution directly calculates weight values ​​based on the first and second pixel blocks, rather than performing alignment processing before calculating the weight values. Therefore, it avoids introducing alignment errors, and the weight values ​​can more accurately and realistically represent the pixel degree between the first and second image content.

[0049] Secondly, in the embodiments of this application, the weight block is calculated based on the similarity between the first pixel block and the second pixel block, rather than based on the similarity of a single pixel. The weight block can more comprehensively consider the image information around the target pixel, so that the value of the weight block can reflect a wider range of image information, thereby helping to improve the denoising effect.

[0050] Thirdly, since the preset step size does not exceed the size of the first pixel block, at least some pixels in the first image can be used multiple times to determine the weight block, which is beneficial to recover more image details during the denoising process, thereby improving the denoising effect.

[0051] To make the above-mentioned objectives, features and beneficial effects of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0052] Reference Figure 2 , Figure 2 This is a flowchart illustrating a multi-frame image denoising method according to an embodiment of this application. The method can be executed by a terminal, such as a mobile phone, computer, or IoT device, but is not limited thereto. The terminal can be equipped with a camera to perform functions such as shooting and recording. For example, the terminal can be equipped with only a single camera, two cameras, or three cameras, but is not limited thereto. The camera configured on the terminal can be a telephoto camera, a wide-angle camera, etc. This embodiment of the invention does not limit the number and type of cameras configured on the terminal. Alternatively, the terminal can be a terminal without a camera, and the terminal can acquire multi-frame images for denoising processing from an external source; this embodiment does not limit this.

[0053] Figure 2 The multi-frame image denoising method shown may include the following steps:

[0054] Step S21: Based on the current target pixel in the first image, determine a first pixel block in the first image. The current target pixel depends on the previous target pixel and a preset step size value. The current target pixel is located at a preset position in the first pixel block.

[0055] Step S22: Determine the second pixel block that matches the first pixel block in each frame of the second image;

[0056] Step S23: For each frame of the second image, determine the weight block of the second image based on the similarity between the second pixel block and the first pixel block, wherein the weight value of each position in the weight block is positively correlated with the similarity.

[0057] Step S24: For each frame of the second image, according to the position of the first pixel block in the first image, the weight block is accumulated to the first weight accumulation map of the second image to update the first weight accumulation map, and the multiplication of the weight block and the second pixel block is accumulated to the first weight pixel multiplication accumulation map of the second image to update the first weight pixel multiplication accumulation map.

[0058] Step S25: If the current target pixel is the last target pixel, then the first weight accumulation map of each frame of the second image is accumulated to obtain the second weight accumulation map, and the first weight pixel multiplication accumulation map of each frame of the second image is accumulated to obtain the second weight pixel multiplication accumulation map.

[0059] Step S26: Obtain the denoised image based on the second weighted pixel multiplication accumulation map and the second weighted accumulation map.

[0060] It is understandable that, in specific implementation, the above method can be implemented using a software program that runs in the processor integrated inside the chip or chip module; or, the method can be implemented using hardware or a combination of hardware and software, such as using a dedicated chip or chip module, or using a dedicated chip or chip module combined with a software program.

[0061] Before executing step S21, multiple frames of images for denoising processing can be acquired. Further, one frame from the multiple frames can be selected as a reference frame. In this embodiment, the image used as the reference frame is designated as the first image, and the other images besides the reference frame are designated as the second images. Thus, the multiple frames can include the first image and at least one second image. In specific implementations, the reference frame can be the first frame of the multiple frames, or it can be the image with the largest image gradient value among the multiple frames.

[0062] Reference Figure 3 , Figure 3 This is a partial flowchart of another multi-frame image denoising method in the embodiments of this application. Figure 3 The method shown may include steps S31 to S33, which may be performed before step S21.

[0063] Step S31: Acquire multiple frames of images.

[0064] Specifically, the terminal can be equipped with a camera. When a shooting command is received, the camera performs a shooting action to capture multiple consecutive images. These multiple consecutive images can be captured using the same exposure value. For example, in practical applications, a user opens an application on the terminal for taking pictures to enter shooting mode. When the user issues a shooting command, the camera performs the shooting action.

[0065] In practice, the number of images acquired can be preset.

[0066] In a non-limiting example, the number of images can depend on the brightness of the shooting scene. The shooting scene can also be referred to as the shooting environment, that is, the environment in which the subject is located. The brighter the shooting scene, the more images can be acquired.

[0067] More specifically, the darker the shooting scene, the stronger the image noise, and the more images need to be acquired to ensure the performance of multi-frame denoising. Conversely, the brighter the shooting scene, the weaker the image noise, and the fewer images can be acquired to improve denoising efficiency.

[0068] In other embodiments, multiple frames of images can also be acquired externally. For example, they can be captured by an external camera and then acquired by the terminal; this embodiment does not limit this.

[0069] Step S32: Remove images that do not meet the preset criteria from the multiple frames of images to obtain the removed images.

[0070] In practice, the quality of multiple captured images often varies. Merging images with significant quality differences for denoising results in poor-quality denoised images, negatively impacting the denoising effect. Therefore, it is necessary to perform frame-by-frame processing on the multiple images.

[0071] Specifically, it can be determined whether each frame of the image meets a preset standard, which can be a preset image quality standard. For example, the preset standard can be a sharpness standard; that is, it can be determined whether the sharpness of each frame of the image is within a preset range. If it is, it can be determined that the preset standard is met; otherwise, it can be determined that the image does not meet the preset standard. In other embodiments, the preset standard can also be other standards used to measure image quality, such as image noise level, etc., and this application does not limit this.

[0072] Furthermore, images that do not meet the preset criteria can be removed to obtain the removed images. That is, the removed images refer to the remaining images after removing images that do not meet the preset criteria from the acquired multi-frame images.

[0073] Step S33: Determine the first image and the second image from the removed images.

[0074] Specifically, one frame can be selected from the removed images as a reference frame and denoted as the first image. Correspondingly, the other images in the removed images besides the first image can be denoted as the second image.

[0075] In a specific example, the first frame of the removed images can be used as the reference frame.

[0076] In another specific example, gradient calculations can be performed on each frame of the removed image, and the frame with the largest sum of pixel gradients can be used as the reference frame.

[0077] Thus, a first image and at least one second image can be obtained for multi-frame denoising. It should be noted that the first and second images can be of the same size.

[0078] Continue to refer to Figure 2 In the specific implementation of step S21, the current target pixel in the first image can first be determined. The current target pixel can be determined based on the previous target pixel and a preset step size value.

[0079] In practice, the pixels in the image can be traversed at intervals of "step," where "step" represents a preset step size. For example, target pixels can be determined sequentially along the row direction or sequentially along the column direction.

[0080] Furthermore, a first pixel block can be determined in the first image based on the current target pixel point in the first image. The size of the first pixel block is m×n, where m and n are both positive integers greater than 1. m and n can be equal or unequal. The current target pixel point can be located at a preset position within the first pixel block. It should be noted that the size of the first pixel block is smaller than the size of the first image.

[0081] Specifically, when determining the target pixels sequentially along the row direction, step < m; when determining the target pixels sequentially along the column direction, step < n. This scheme allows at least some pixels in the first pixel to participate in the weight value calculation multiple times.

[0082] In a non-restrictive example, the preset step size is 1, which means that each pixel in the first image is taken as the current target pixel in turn.

[0083] Furthermore, the preset position can be the center position. That is, the current target pixel is located at the center of the first pixel block. For example, if the size of the first pixel block is 3×3, then the preset position is in the 2nd row and 2nd column. Or, for example, if the size of the first pixel block is 5×5, the preset position is in the 3rd row and 3rd column.

[0084] Alternatively, the preset position can be any position other than the center position, and this application embodiment does not impose any limitations. For example, the preset position can be the first row and first column, that is, the current target pixel value can be located in the first row and first column of the first pixel block.

[0085] Therefore, as the target position is updated, the first pixel block is also updated. After determining the current target position, the first pixel block can be determined in the first image based on the preset position and the current target position.

[0086] It should be noted that, before executing step S21, boundary expansion processing can also be performed on the first image and the second image. The specific method for boundary expansion processing can be any existing suitable method, and this embodiment does not limit it.

[0087] In a specific implementation of step S22, a second pixel block that matches the first pixel block can be determined in each frame of the second image.

[0088] In practice, the pixel corresponding to the current target pixel can be determined in the second image of each frame and recorded as the matching pixel.

[0089] In one embodiment of this application, if the current target pixel is the pixel in the i-th row and j-th column of the first image, then the matching pixel can be the pixel in the i-th row and j-th column of the second image. That is, the matching pixel refers to the pixel in the second image that is at the same position as the current target pixel.

[0090] In another embodiment of this application, the first image and each frame of the second image each include Q M×N image blocks. The current target pixel is the pixel in the i-th row and j-th column of the q-th image block in the first image. Then the matching pixel is the pixel in the i-th row and j-th column of the matching image block of the q-th image block in the second image.

[0091] Specifically, the first image and each frame of the second image can be divided into Q equal-sized, non-overlapping image blocks, and each pixel in the first image belongs to one and only one image block. For the q-th image block in the first image, image block matching can be performed to determine the matching image block of the q-th image block. More specifically, the image block with the highest similarity to the q-th image block can be determined in each frame of the second image, and denoted as the matching image block of the q-th image block.

[0092] In a specific implementation, for each frame of the second image, the distances between the Q image blocks and the q-th image block can be calculated separately, and the image block with the smallest distance is taken as the matching image block of the q-th image block. Various existing methods can be used to calculate the distances between image blocks, and this embodiment does not limit this approach. For example, the distances between image blocks can be represented by calculating the L1 norm, or they can be represented by calculating the L2 norm.

[0093] In a non-limiting example, a multi-resolution image pyramid approach can be used for image patch matching. Optionally, sub-pixel processing can be performed during image patch matching to improve the accuracy of image patch matching and reduce computational cost.

[0094] Furthermore, the pixel in the i-th row and j-th column of the matching image block of the q-th image block in each frame of the second image can be used as the matching pixel.

[0095] Compared to directly using the pixel in the second image that is at the same position as the current target pixel as the matching pixel, the method of first performing image block matching and then determining the matching pixel in the matched image block is beneficial to improving the image denoising effect, and can ensure good denoising performance even when there is local motion in the image.

[0096] Furthermore, the second pixel block can be determined based on the position of the matching pixel. The size of the second pixel block is the same as the size of the first pixel block. The matching pixel is located within the second pixel block, and its position within the second pixel block can be related to a preset position. That is, the position of the matching pixel in the second pixel block is related to the position of the current target pixel in the first pixel block.

[0097] In a specific example, a single second pixel block is determined in each frame of the second image. That is, there is a one-to-one correspondence between the first pixel block and the second pixel block in each frame of the second image.

[0098] More specifically, the matching pixel is located at a preset position in the second pixel block. That is, the position of the matching pixel in the second pixel block can be the same as the position of the current target pixel in the first pixel block. For example, if the current target pixel is located at the center of the first pixel block, the matching pixel is located at the center of the second pixel block.

[0099] In another specific example, multiple second pixel blocks are determined in each frame of the second image. That is, there is a one-to-many correspondence between the first pixel block and the second pixel block in each frame of the second image.

[0100] More specifically, each frame of the second image contains multiple second pixel blocks, each of which may include a single second center pixel block and at least one second neighboring pixel block. Matching pixels are located at a preset position within the second center pixel block, and matching pixels are located at the periphery of the second neighboring pixel blocks. The periphery can refer to any position other than the preset position.

[0101] More specifically, when the current target pixel is located at the center of the first pixel block, the matching pixel can be located at the center of the second center pixel block, and the matching pixel block can be located around the second neighboring pixel block. This allows multiple second pixel blocks to be determined in each frame of the second image.

[0102] Taking the first pixel block size of 3×3 as an example, the preset position is the center position, that is, the preset position is the second row and the second column. The surrounding positions can include one or more of the following: the first row and the second column, the second row and the first column, the third row and the second column, and the second row and the third column.

[0103] More intuitively, the current target pixel can be represented as: ref_b(i, j), the matching pixel can be represented as: cur_b(i, j), the first pixel block can be represented as: ref_b_patch_c = ref_b(i-1: i+1, j-1: j+1), the second center pixel block can be represented as: cur_b_patch_c = cur_b(i-1: i+1, j-1: j+1), and the second neighboring pixel block can include one or more of the following four:

[0104] cur_b_patch_l=cur_b(i-1:i+1,j-2:j)

[0105] cur_b_patch_r=cur_b(i-1:i+1,j:j+2)

[0106] cur_b_patch_u=cur_b(i-2:i,j-1:j+1)

[0107] cur_b_patch_d=cur_b(i:i+2,j-1:j+1)

[0108] Based on the above, one or more second pixel blocks corresponding to the first pixel block can be determined in each frame of the second image.

[0109] In a specific implementation of step S23, for each second pixel block in each frame of the second image, a weight block for the second pixel block can be determined based on the similarity between the second pixel block and the first pixel block. The size of the weight block can be the same as the size of the first pixel block.

[0110] Specifically, the value at each position in the weight block is the weight value, and the weight value at each position can be the same. That is, the weight value at each position in the weight block of the second pixel block represents the similarity between the entire second pixel block and the first pixel block. The weight value and the similarity are positively correlated; that is, the greater the similarity, the larger the weight value.

[0111] In a specific example, the weight value of the weight block can be calculated using the following formula:

[0112]

[0113] Among them, W p Here, σ is the weight value in the weighted block, sigma is a preset parameter value, and diff is the difference between the first and second pixel blocks. The preset parameter value can be pre-set according to actual needs, and changing the preset parameter value can control the rate at which the weight value decays due to the difference between pixel blocks. The difference between the first and second pixel blocks can refer to the distance between the first and second pixel blocks, which can be represented by the L1 norm or L2 norm. It is understood that the larger the diff value, the smaller the similarity between the first and second pixel blocks, and the smaller the weight value.

[0114] It should be noted that for each frame of the second image, if there are multiple second pixel blocks, a weight block is calculated for each second pixel block separately. That is, for each frame of the second image, the weight block corresponding to each second pixel block is calculated. In other words, there is a one-to-one correspondence between second pixel blocks and weight blocks.

[0115] In the specific implementation of step S24, for each second pixel block of each frame of the second image, the weight block can be accumulated to the weight accumulation map of the second image according to the position of the first pixel block in the first image, so as to update the first weight accumulation map of the second image; and the multiplication of the weight block and the second pixel block can be accumulated to the first weight pixel multiplication accumulation map of the second image, so as to update the first weight pixel multiplication accumulation map of the second image.

[0116] On the one hand, the second image and the first weight are accumulated. Figure 1 The first weighted cumulative image is obtained by accumulating the weighted blocks of the second pixel block, and the two pixels are in one-to-one correspondence and have the same size.

[0117] Specifically, the initial value at each position in the first weighted cumulative image can be 0. Whenever a weight block of the second pixel block is obtained, the weight block is added to the first weighted cumulative image based on the position of the first pixel block in the first image, thereby updating the first weighted cumulative image of the second image. More specifically, the weight block is added to the same position in the first weighted cumulative image based on the position of the first pixel block in the first image. Thus, as the target pixel is updated, the first weighted cumulative images of each second image are continuously updated. It should be noted that the "cumulative" referred to in this embodiment means the addition of values ​​at the same position.

[0118] On the other hand, the second image and the first weighted pixel multiplication accumulation are added together. Figure 1The first weighted pixel multiplication summation image, with a one-to-one correspondence and identical size, is obtained by multiplying and summing the second pixel block and the weighted blocks of the second pixel block. Specifically, the product of the second pixel block and the weighted blocks of the second pixel block is the result of multiplying the pixel value at each position in the second pixel block with the weight value at the same position in the weighted block. In other words, the product of the second pixel block and the weighted blocks of the second pixel block is the result of multiplying the pixel value at each position in the second pixel block with the weight value at the same position in the weighted block.

[0119] Specifically, the initial values ​​at each position in the first weighted pixel multiplication accumulation map can be 0. Whenever the weight block of the second pixel block is obtained, the product of the second pixel block and its weight block is calculated. Then, based on the position of the first pixel block in the first image, the product of the second pixel block and its weight block is added to the same position in the first weighted pixel multiplication accumulation map. Thus, as the target pixels are continuously updated, the first weighted pixel multiplication accumulation map of each second image is also continuously updated.

[0120] In a specific implementation, if each frame of the second image includes only a single second pixel block, then the first weighted cumulative image and the first weighted pixel multiplication cumulative image of the second image can be updated only according to the weight block of that second pixel block. That is, when the target pixel is updated once, the first weighted cumulative image and the first weighted pixel multiplication cumulative image of each frame of the second image are updated once respectively.

[0121] If each frame of the second image includes multiple second pixel blocks, and correspondingly, there are multiple weight blocks, and the second pixel blocks and weight blocks correspond one-to-one, then the first weight accumulation map and the first weight pixel multiplication accumulation map of the second image can be updated sequentially according to the weight blocks of each second pixel block.

[0122] More specifically, the weight blocks of each second pixel block can be sequentially accumulated into the first weight accumulation map of the second image, and the product of each second pixel block and its weight block can be sequentially accumulated into the first weight pixel product accumulation map. Thus, for each target pixel point updated, the first weight accumulation map and the first weight pixel product accumulation map of each frame of the second image are updated multiple times, and the number of updates is the same as the number of second pixel blocks in the second image.

[0123] It should be noted that when the weight blocks are accumulated in the first weight accumulation image, the accumulation is based on the position of the first pixel block in the first image. That is, the position of the weight blocks accumulated in the first weight accumulation image is the same as the position of the first pixel block in the first image. Correspondingly, the position of the product of the second pixel block and the weight blocks accumulated in the first weight pixel product accumulation image is the same as the position of the first pixel block in the first image.

[0124] Therefore, when each frame of the second image includes multiple second pixel blocks, the weight blocks of each second pixel block are all accumulated to the same position in the first weight accumulation map (that is, the position of the first pixel block in the first image), and the product of each second pixel block and its corresponding weight block is also accumulated to the same position in the first weight pixel multiplication accumulation map.

[0125] In the specific implementation of step S25, it can be determined whether the current target pixel is the last target pixel. More specifically, the next target pixel can be determined based on the current target pixel and a preset step size value. If the next target pixel exceeds the range of the first image, then the current target pixel can be determined as the last target pixel.

[0126] If the current target pixel is the last target pixel, then the update of the first weighted cumulative image and the first weighted pixel multiplication cumulative image of each frame of the second image can be determined. Further, the first weighted cumulative images of each frame of the second image can be accumulated to obtain the second weighted cumulative image, and the first weighted pixel multiplication cumulative images of each frame of the second image can be accumulated to obtain the second weighted pixel multiplication cumulative image.

[0127] In other words, the second weighted cumulative image is obtained by summing the first weighted cumulative images corresponding to each frame of the second image; that is, there is a many-to-one correspondence between the second images and the second weighted cumulative image. Correspondingly, the second weighted pixel multiplication cumulative image is obtained by summing the first weighted pixel multiplication cumulative images corresponding to each frame of the second image; that is, there is also a many-to-one correspondence between the second images and the second weighted pixel multiplication cumulative image. The second weighted cumulative image and the second weighted pixel multiplication cumulative image have the same size.

[0128] If the current target pixel is not the last target pixel, the next target pixel can be taken as the current target pixel, and the process returns to step S21. Then, steps S21 to S25 are executed until the current target pixel is the last target pixel.

[0129] Based on the above, by determining the target pixel points in sequence, the second weighted cumulative image and the second weighted pixel multiplication cumulative image are finally obtained.

[0130] In the specific implementation of step S26, the denoised image can be obtained based on the second weighted pixel multiplication accumulation map and the second weighted accumulation map.

[0131] Specifically, the first image may also have a corresponding weight map. The weight values ​​at each position in the weight map of the first image may be preset values, which may be calculated by an appropriate method. This embodiment does not limit the calculation method of the weight values ​​in the weight map of the first image.

[0132] Furthermore, the product of the first image and its weight map can be calculated to obtain the weighted pixel product map of the first image. That is, the pixel value at each position in the first image is multiplied by the weight value at the same position in the weight map to obtain the weighted pixel product map of the first image.

[0133] Furthermore, the second weighted pixel multiplication cumulative image and the weighted pixel multiplication image of the first image can be accumulated to obtain the third weighted pixel multiplication cumulative image, and the second weighted cumulative image and the weighted image of the first image can be accumulated to obtain the third weighted cumulative image.

[0134] Furthermore, the value of each position in the third weighted pixel multiplication accumulator can be divided by the value of the corresponding position in the third weighted accumulator to obtain the pixel value of the corresponding position in the denoised image, thus obtaining the denoised image.

[0135] Reference Figure 4 , Figure 4 This is a schematic diagram of the structure of a multi-frame image denoising device according to an embodiment of this application. Figure 4 The apparatus shown may include:

[0136] The first determining module 41 is used to determine a first pixel block in the first image based on the current target pixel in the first image. The current target pixel depends on the previous target pixel and a preset step value. The current target pixel is located at a preset position of the first pixel block. The size of the first pixel block is m×n. The preset step value step satisfies: step < m and / or step < n.

[0137] The second determining module 42 is used to determine a second pixel block that matches the first pixel block in each frame of the second image;

[0138] The third determining module 43 is used to determine the weight block of the second image for each frame of the second image based on the similarity between the second pixel block and the first pixel block, wherein the weight value of each position in the weight block is positively correlated with the similarity.

[0139] The first calculation module 44 is configured to, for each frame of the second image, add the weight block to the first weight accumulation map of the second image according to the position of the first pixel block in the first image, so as to update the first weight accumulation map, and add the multiplication of the weight block and the second pixel block to the first weight pixel multiplication accumulation map of the second image, so as to update the first weight pixel multiplication accumulation map.

[0140] The second calculation module 45 is used to accumulate the first weight accumulation map of each frame of the second image to obtain the second weight accumulation map if the current target pixel is the last target pixel, and to accumulate the first weight pixel multiplication accumulation map of each frame of the second image to obtain the second weight pixel multiplication accumulation map.

[0141] The third calculation module 46 is used to obtain the denoised image based on the second weighted pixel multiplication accumulation map and the second weighted accumulation map.

[0142] For more information on the working principle, working method, and beneficial effects of the multi-frame image denoising device in the embodiments of this application, please refer to the relevant description of the multi-frame image denoising method above, which will not be repeated here.

[0143] In practice, Figure 4 The multi-frame image denoising device shown can correspond to a chip with image processing function in a terminal, or to a chip with image processing function, such as an image signal processing (ISP) chip; or to a chip module with image processing function, or to a terminal.

[0144] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it performs the steps of the multi-frame image denoising method described above. The storage medium may include ROM, RAM, disk, or optical disk, etc. The storage medium may also include non-volatile memory or non-transitory memory, etc.

[0145] This application also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it performs the steps of the multi-frame image denoising method described above. The terminal includes, but is not limited to, terminal devices such as mobile phones, computers, and tablet computers.

[0146] This application also provides a chip that can be used to perform the above-described multi-frame image denoising method. Alternatively, the chip may include... Figure 4 The illustrated multi-frame image denoising device. In a specific example, the chip may be an image signal processing (ISP) chip.

[0147] This application also provides a chip module that can be used to perform the above-described multi-frame image denoising method. Alternatively, the chip module may include... Figure 4 The multi-frame image denoising device shown is illustrated.

[0148] It should be understood that in the embodiments of this application, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0149] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0150] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0152] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or in a combination of hardware and software functional units. For example, for various devices or products applied to or integrated into a chip, each module / unit can be implemented using hardware such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware such as circuits; for various devices or products applied to or integrated into a chip module, each module / unit can be implemented using hardware such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware such as circuits. The components can be implemented using software programs that run on the processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its components / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or in different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.

[0153] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0154] In this application's embodiments, "multiple" refers to two or more. The descriptions of "first," "second," etc., appearing in this application's embodiments are merely illustrative and for distinguishing the described objects; they do not indicate any order and do not imply a specific limitation on the number of devices in this application's embodiments, nor do they constitute any limitation on the embodiments of this application. Although this application discloses the above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

[0155] While this application discloses the above information, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application shall be determined by the scope defined in the claims.

Claims

1. A method for denoising multi-frame images, characterized in that, The multi-frame image includes a first image and at least one second image, and the method includes: Based on the current target pixel in the first image, a first pixel block is determined in the first image. The current target pixel depends on the previous target pixel and a preset step size value. The current target pixel is located at a preset position in the first pixel block, and the size of the first pixel block is [missing value]. The preset step size value step satisfy: and / or ; In each frame of the second image, a second pixel block that matches the first pixel block is determined; For each frame of the second image, a weight block of the second image is determined based on the similarity between the second pixel block and the first pixel block, wherein the weight value of each position in the weight block is positively correlated with the similarity. For each frame of the second image, based on the position of the first pixel block in the first image, the weight block is accumulated to the first weight accumulation map of the second image to update the first weight accumulation map, and the product of the weight block and the second pixel block is accumulated to the first weight pixel product accumulation map of the second image to update the first weight pixel product accumulation map. If the current target pixel is the last target pixel, then the first weighted cumulative image of each frame of the second image is accumulated to obtain the second weighted cumulative image, and the first weighted pixel multiplication cumulative image of each frame of the second image is accumulated to obtain the second weighted pixel multiplication cumulative image. The denoised image is obtained by combining the second weighted pixel multiplication accumulation map and the second weighted accumulation map. The step of obtaining the denoised image based on the second weighted pixel multiplication accumulation image and the second weighted accumulation image includes: The second weighted pixel multiplication summation map and the weighted pixel multiplication map of the first image are summed to obtain the third weighted pixel multiplication summation map, wherein the weighted pixel multiplication map of the first image is obtained by calculating the product of the first image and its weight map; The second weighted cumulative image and the weighted image of the first image are summed to obtain the third weighted cumulative image; Divide the value at each position in the third weighted pixel multiplication accumulator by the value at the corresponding position in the third weighted accumulator to obtain the pixel value at the corresponding position in the denoised image.

2. The multi-frame image denoising method according to claim 1, characterized in that, The preset step size is 1.

3. The multi-frame image denoising method according to claim 1, characterized in that, The preset position is the center position.

4. The multi-frame image denoising method according to claim 1, characterized in that, The second pixel block includes a second center pixel block and a second neighboring pixel block. Determining the second pixel block that matches the first pixel block in each frame of the second image includes: Based on the current target pixel, determine the pixel corresponding to the current target pixel in each frame of the second image, and record it as the matching pixel; Based on the matching pixel points, the second center pixel block and the second neighboring pixel block are determined; Wherein, the matching pixel is located at a preset position in the second center pixel block, and the matching pixel is located at a peripheral position in the second neighboring pixel block, where the peripheral position refers to a position other than the preset position.

5. The multi-frame image denoising method according to claim 4, characterized in that, The size of the first pixel block is The preset position is the 2nd row and 2nd column, and the surrounding positions include any one or more of the following: the 1st row and 2nd column, the 2nd row and 1st column, the 3rd row and 2nd column, and the 2nd row and 3rd column.

6. The multi-frame image denoising method according to claim 1, characterized in that, Both the first image and the at least one frame of the second image include Q indivual Image block, the current target pixel is the first pixel in the first image. q The first image patch i Line number j The columns of pixels, in each frame of the second image, determine the second pixel block that matches the first pixel block, including: In the second image of each frame Q The image block is determined to be related to the first one. q The image patch with the highest similarity to the nth image patch is denoted as the nth image patch. q Matching image patches for each image patch; According to the first q In the matching image patch of the nth image patch i Line number j The pixels in the column are used to determine the second pixel block in each frame of the second image; Among them, the first in the matching image block i Line number j The pixels in the column are located at a preset position in the second pixel block. , , , Q , M , N It is a positive integer.

7. The multi-frame image denoising method according to claim 1, characterized in that, The weight values ​​in the weight block are calculated and determined according to the following formula: in, The weight values ​​in the weight block, For preset parameter values, This is the difference value between the second pixel block and the first pixel block.

8. The multi-frame image denoising method according to claim 1, characterized in that, Before determining the first pixel block in the first image based on the current target pixel point in the first image, the method further includes: Images that do not meet the preset criteria are removed from the multiple frames of images to obtain the removed images, which include the first image and the at least one second frame.

9. A multi-frame image denoising device, characterized in that, The multi-frame image includes a first image and at least one second image, and the device includes: A first determining module is configured to determine a first pixel block in the first image based on a current target pixel in the first image. The current target pixel depends on a previous target pixel and a preset step size. The current target pixel is located at a preset position within the first pixel block, and the size of the first pixel block is [missing information]. The preset step size value step satisfy: and / or ; The second determining module is used to determine a second pixel block that matches the first pixel block in each frame of the second image; The third determining module is used to determine the weight block of the second image for each frame of the second image based on the similarity between the second pixel block and the first pixel block, wherein the weight value of each position in the weight block is positively correlated with the similarity. The first calculation module is configured to, for each frame of the second image, add the weight block to the first weight accumulation map of the second image according to the position of the first pixel block in the first image, so as to update the first weight accumulation map, and add the multiplication of the weight block and the second pixel block to the first weight pixel multiplication accumulation map of the second image, so as to update the first weight pixel multiplication accumulation map. The second calculation module is used to accumulate the first weight accumulation map of each frame of the second image to obtain the second weight accumulation map if the current target pixel is the last target pixel, and to accumulate the first weight pixel multiplication accumulation map of each frame of the second image to obtain the second weight pixel multiplication accumulation map. The third calculation module is used to obtain the denoised image based on the second weighted pixel multiplication accumulation map and the second weighted accumulation map; The third calculation module performs the following steps: The second weighted pixel multiplication summation map and the weighted pixel multiplication map of the first image are summed to obtain the third weighted pixel multiplication summation map, wherein the weighted pixel multiplication map of the first image is obtained by calculating the product of the first image and its weight map; The second weighted cumulative image and the weighted image of the first image are summed to obtain the third weighted cumulative image; Divide the value at each position in the third weighted pixel multiplication accumulator by the value at the corresponding position in the third weighted accumulator to obtain the pixel value at the corresponding position in the denoised image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the multi-frame image denoising method according to any one of claims 1 to 8.

11. A terminal comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the multi-frame image denoising method according to any one of claims 1 to 8.

12. A chip, characterized in that, The chip is used to perform the steps of the multi-frame image denoising method according to any one of claims 1 to 8.

13. A chip module, characterized in that, The chip module is used to perform the steps of the multi-frame image denoising method according to any one of claims 1 to 8.

Images