Noise reduction method and device, storage medium and electronic equipment

Abstract

The invention provides a noise reduction method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining a first image collected by an image sensor in a first mode and a second image collected by an image sensor in a second mode; performing clustering on each pixel region in the first image to obtain a first clustering result; in response to determining that the first image does not generate pixel position offset relative to the second image, acquiring a corresponding relationship between each pixel point in the first image and each pixel point in the second image; obtaining a second clustering result of each pixel region in the second image based on the corresponding relationship and the first clustering result; and carrying out noise reduction on the second image based on the second clustering result. According to the method provided by the embodiment of the invention, the second image can be subjected to noise reduction, and a relatively accurate data truth value can be obtained.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a noise reduction method, apparatus, storage medium, and electronic device. Background Technology

[0002] The true value of data refers to the actual, accurate, and unobservable numerical value corresponding to the measured variable under a specific measurement or experimental condition. In the field of image processing, the true value specifically refers to the original grayscale value or color information that a pixel in an image should theoretically possess—a value free of noise and distortion.

[0003] In practical applications, due to the influence of various complex factors such as measurement errors, equipment performance limitations, and environmental noise, it is difficult to directly obtain the ideal "true value" that is unaffected by any noise.

[0004] Taking the remosaic mode in an image sensor as an example, this mode aims to convert raw data from a Bayer array into a full-color image. During this process, due to inconsistencies in hardware response, dynamic changes in lighting conditions, and other uncontrollable factors, the image sensor is highly susceptible to noise when capturing the color information of each pixel. This noise pollution significantly hinders the accurate reconstruction of color information for each pixel, resulting in full-color images generated through the remosaic mode often exhibiting high noise levels. Therefore, a solution is needed. Summary of the Invention

[0005] In view of this, the present disclosure provides a noise reduction method, apparatus, storage medium and electronic device for denoising images, which helps to obtain more accurate data truth values.

[0006] According to a first aspect of the present disclosure, a noise reduction method is provided, the method comprising:

[0007] Acquire a first image captured by an image sensor in a first mode and a second image captured in a second mode;

[0008] Clustering is performed on each pixel region in the first image to obtain a first clustering result; the clustering accuracy of the first clustering result is higher than the clustering accuracy of clustering each pixel region in the second image.

[0009] In response to determining that the first image has no pixel position offset compared to the second image, the correspondence between each pixel in the first image and the second image is obtained;

[0010] Based on the correspondence and the first clustering result, a second clustering result is obtained for each pixel region in the second image;

[0011] The second image is denoised based on the second clustering result.

[0012] According to a second aspect of the present disclosure, a noise reduction apparatus is provided, the apparatus comprising:

[0013] The first acquisition module is used to acquire a first image acquired by the image sensor in a first mode and a second image acquired in a second mode;

[0014] The first clustering module is used to cluster each pixel region in the first image to obtain a first clustering result; the clustering accuracy of the first clustering result is higher than that of the clustering result of clustering each pixel region of the second image.

[0015] The second acquisition module is used to acquire the correspondence between each pixel in the first image and the second image in response to determining that the first image has not produced a pixel position offset relative to the second image;

[0016] The second clustering module is used to obtain a second clustering result for each pixel region in the second image based on the correspondence and the first clustering result;

[0017] A noise reduction module is used to reduce noise in the second image based on the second clustering result.

[0018] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0019] processor;

[0020] Memory used to store processor-executable instructions;

[0021] The processor is configured to implement the steps of any of the noise reduction methods of the first aspect by running the executable instructions.

[0022] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the noise reduction methods described in the first aspect.

[0023] The technical solutions provided in this disclosure may have the following beneficial effects:

[0024] By acquiring first and second images obtained under different modes, the first image can be clustered to obtain a relatively accurate first clustering result. Then, based on the one-to-one correspondence between each pixel in the first and second images and the first clustering result, a second clustering result for the second image is obtained. Compared with the scheme of directly clustering the second image, this scheme can guarantee the clustering accuracy of the second clustering result, thus helping to accurately reduce noise in the second image based on the second clustering result.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0027] Figure 1 This disclosure is a schematic diagram illustrating an image processing method according to an exemplary embodiment;

[0028] Figure 2 This is another image processing schematic diagram illustrated according to an exemplary embodiment of the present disclosure;

[0029] Figure 3 This is a flowchart illustrating a noise reduction method according to an exemplary embodiment of the present disclosure;

[0030] Figure 4a This is a schematic diagram of a noise reduction method according to an exemplary embodiment of the present disclosure;

[0031] Figure 4b This is a schematic image illustrating another noise reduction method according to an exemplary embodiment of the present disclosure;

[0032] Figure 5 This is a schematic image illustrating another noise reduction method according to an exemplary embodiment of the present disclosure;

[0033] Figure 6 This is a schematic diagram of the structure of a noise reduction device according to an exemplary embodiment of the present disclosure;

[0034] Figure 7 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0036] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0037] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0038] Image sensors typically have multiple data output modes, the most common being binning mode and remosaic mode. These two modes are used for different optimization services.

[0039] Specifically, binning mode enhances signal strength, effectively reduces noise, and may increase frame rate by merging signals from adjacent pixels. This is crucial for improving image quality in low-light environments and capturing dynamic scenes.

[0040] Figure 1 This disclosure illustrates an image processing schematic diagram according to an exemplary embodiment, such as... Figure 1 As shown, in binning mode:

[0041] The 16 pixels on the left are merged into 4 pixels, meaning that every 4 pixels are merged into 1 pixel.

[0042] The remosaic mode focuses on reconstructing a complete full-color image from the raw data captured by the Bayer array from the image sensor. This process involves interpolation calculations of individual color information at each pixel, aiming to restore the red, green, and blue color components that each pixel should have, thereby generating a high-resolution color image. However, in practice, because it relies on interpolation algorithms to fill in missing color information, especially under complex lighting conditions and in situations requiring high detail reproduction, it may introduce additional noise components, thus affecting the quality of the output image and the accuracy of the data.

[0043] The refactoring process can be referenced. Figure 2 The image on the left shows the original pixel layout of 4 cells x 1 chip. "4 cells x 1 chip" refers to a pixel array layout where each chip has four pixel units, each containing one green and one blue sensor. This layout is commonly referred to as "4 cell x 1 chip" or "4C1P". The image on the right is a reconstruction of the pixel arrangement on the left into a traditional Bayer layout. This reconstruction involves complex image signal processing, including filtering, noise reduction, and interpolation. While this allows the image to contain more information, it also increases the data volume of a single image and may introduce more noise. This results in remosaic data having a larger data volume per frame and higher noise levels compared to binning data, making the data more susceptible to the influence of the current noise level. These characteristics make it difficult to obtain the true value of the data in remosaic mode.

[0044] Therefore, a solution is needed to reduce image noise and obtain the true value of image data.

[0045] The solution is as follows:

[0046] First, considering that the second image in the second mode (assuming it is an image in remosaic mode) usually has high noise, it is necessary to denoise the second image in order to obtain the true data value. Therefore, this disclosure proposes to first cluster similar pixel regions in the second image, and then denoise based on the clustering results, which helps to obtain a more accurate true data value.

[0047] However, considering that noise in the second image can greatly affect the clustering results of the second image, and the clustering accuracy of the clustering results can affect the noise reduction results, it is necessary to improve the clustering accuracy of the second image.

[0048] In this case, a first mode (let's assume it's binning mode) is introduced. Considering that the first image acquired using the first mode usually has a low noise level, the first clustering result obtained by directly clustering the first image is often quite accurate.

[0049] Thus, without changing the orientation of the image acquisition sensor, the system switches to the second mode and acquires the second image using this mode. At this point, the first image does not differ from the second image in terms of pixel location; therefore, each pixel in the first and second images corresponds one-to-one. Since the first image already contains the first clustering result, the pixels in the second image can be directly clustered based on the positions of the pixels in the first clustering result, resulting in a more accurate second clustering result.

[0050] In this way, when denoising the second image based on the second clustering result, a more accurate true value of the data can be obtained.

[0051] The following is a detailed introduction:

[0052] Figure 3 This is a flowchart illustrating a noise reduction method according to an exemplary embodiment of the present disclosure, such as... Figure 3 As shown, the method includes the following steps:

[0053] Step 301: Acquire a first image captured by the image sensor in a first mode and a second image captured in a second mode.

[0054] For example, the image sensor can activate a first mode, a second mode, or other supported modes. The image sensor employs a fixed acquisition posture when acquiring the first and second images, ensuring that the first and second images capture the same content. Whether it's the first or second image, their pixel layout and corresponding physical spatial regions remain unchanged, ensuring that the content of each acquired image is strictly corresponding.

[0055] This means that in the first and second images, the same pixel on the image sensor will capture information about the same part of the scene, rather than capturing slightly overlapping or misaligned parts due to sensor movement. As a result, the two images are perfectly aligned at the pixel level, allowing for direct comparison, overlay, or other forms of conventional image processing without the need to consider pixel position matching. This is crucial for applications that rely on consistent comparison of consecutive images (such as continuous video recording and burst shooting), where subsequent images typically require no pixel-level offset relative to the previous image.

[0056] In this context, the first image acquired using the first mode typically has a lower noise level compared to the second image acquired using the second mode, where the second image is the image to be denoised. For example, the first mode could be a binning mode, and the second mode could be a remosaic mode.

[0057] Step 302: Cluster each pixel region in the first image to obtain a first clustering result; the clustering accuracy of the first clustering result is higher than the clustering accuracy of clustering each pixel region in the second image.

[0058] Here, if the goal is simply to reduce noise in the first image, we can first find similar pixel regions in the first image, then aggregate these similar regions together, and then reduce the noise of the image while preserving the details and structural information of the image through collaborative filtering, similar to the processing ideas of block matching and 3D filtering.

[0059] In this process, the process of "finding similar pixel regions in the first image and then aggregating these similar regions together" is the process of obtaining the first clustering result by the above clustering.

[0060] For example, when the first mode is binning mode and the second mode is remosaic mode, the clustering accuracy obtained by directly clustering the first image (the first clustering result) is higher than the clustering accuracy obtained by directly clustering the second image. Alternatively, when the noise level of the first image is lower than the noise level of the second image, the clustering accuracy obtained by directly clustering the first image (the first clustering result) is also higher than the clustering accuracy obtained by directly clustering the second image.

[0061] Therefore, it is necessary to first cluster the first image to obtain the first clustering result, and then indirectly cluster the second image based on the first clustering result to obtain a more accurate second clustering result. See the following steps for details.

[0062] Step 303: In response to determining that the first image has not produced a pixel position offset compared to the second image, obtain the correspondence between each pixel point in the first image and the second image.

[0063] There is no pixel position shift between the first and second images, indicating that there is no pixel-level positional shift between them. In this case, every pixel in the first image can be found at the same position in the second image. That is, there is a one-to-one correspondence between the pixels in the first and second images, thus the correspondence between the individual pixels can be determined.

[0064] Figure 4a This is a schematic diagram of a noise reduction method according to an exemplary embodiment of the present disclosure; Figure 4b This is an image illustration of another noise reduction method according to an exemplary embodiment of the present disclosure (wherein, Figure 4b It has a very high noise level. Figure 4a The first image is shown. Figure 4bThe image shown is the second image. The pixel correspondence between the first and second images is as follows (this is only illustrative; in reality, an image contains many pixels, but for simplicity, we assume an image contains only six pixels):

[0065] Pixel 401 of the first image corresponds to pixel 411 of the second image; pixel 402 of the first image corresponds to pixel 412 of the second image; pixel 403 of the first image corresponds to pixel 413 of the second image; pixel 404 of the first image corresponds to pixel 414 of the second image; pixel 405 of the first image corresponds to pixel 415 of the second image; and pixel 406 of the first image corresponds to pixel 416 of the second image.

[0066] Step 304: Based on the correspondence and the first clustering result, a second clustering result is obtained for each pixel region in the second image.

[0067] This is still combined Figure 4a and Figure 4b Analyzing the first image, if we cluster the pixels, we can divide each pixel into different clusters (each pixel is assigned to a separate cluster), resulting in the first clustering result. Then, based on the similarity features of the pixels, pixels 401 and 402 can be clustered into one cluster, and pixels 404 and 405 into another. Pixels 403 and 406 are not analyzed; their fate—whether they form a single cluster, a separate cluster, or are assigned to another existing cluster—can be determined based on the specific clustering algorithm used.

[0068] Here, we assume the first clustering result is:

[0069] Pixels 401 and 402 are clustered into cluster 1, pixel 403 is clustered into cluster 2, pixels 404 and 405 are clustered into cluster 3, and pixel 406 is clustered into cluster 4.

[0070] Based on the correspondence in step 303 above and the first clustering result above, the following second clustering result can be obtained:

[0071] Pixels 411 and 412 are clustered into cluster one, pixel 413 into cluster two, pixels 414 and 415 into cluster three, and pixel 416 into cluster four. Thus, the second clustering result obtained through the above steps is more accurate than directly clustering the second image.

[0072] Step 305: Denoise the second image based on the second clustering result.

[0073] After obtaining the second clustering result through the above scheme, the second image is denoised using the second clustering result. Here, Wiener filtering and other methods can be introduced to denoise, so as to obtain the denoised second image, which is the true data value of the image captured in the second mode.

[0074] Optionally, the resolution of the first image is different from the resolution of the second image. In this case, step 303, in response to determining that the first image has not produced a pixel position offset relative to the second image, includes the following steps:

[0075] The resolution of the first image is set to be the same as that of the second image; it is determined that the first image and the second image do not produce pixel position offsets at the same resolution.

[0076] Since the first mode and the second mode are different output modes, the first image acquired by the first mode and the second image acquired by the second mode typically have different resolutions. In this case, to determine whether the first image and the second image have pixel position offsets, they need to be set to the same resolution first.

[0077] For example, when the resolution of the first image is higher than that of the second image, in order to unify them to the same resolution, we can perform a downsampling operation on the first image to reduce its resolution, or perform an upsampling operation on the second image to increase its resolution. In this way, we can obtain two images with the same resolution.

[0078] For example, if the first mode is binning mode and the second mode is remosaic mode, the resolution of the second image acquired in the second mode is higher than the resolution of the first image acquired in the first mode. To determine if there is a pixel-level positional shift between the first and second images, the second image can be downsampled to obtain an image at a target resolution; the target resolution is the resolution of the first image. Then, it is determined that the first image has not experienced a pixel-level positional shift compared to the image at the target resolution.

[0079] Optionally, step 305 involves denoising the second image based on the second clustering result, including the following steps:

[0080] The pixel regions in each cluster of the second clustering result are denoised; the denoised pixel regions are then restored to their original positions in the second image to obtain the denoised second image.

[0081] Here, noise and non-noise can be determined based on the similarity features of pixels in each cluster, and the denoised pixel regions (pixels) can be obtained.

[0082] Alternatively, noise reduction can be achieved using Wiener filtering. Here, the first clustering result can also be used as the input image for Wiener filtering to reduce noise in the second clustering result.

[0083] After denoising the pixels in each cluster, the pixels are reset to their original positions to obtain the denoised second image, which is the true data value.

[0084] Optionally, the first mode is the binning mode of the image sensor.

[0085] Step 301, acquiring the first image captured by the image sensor in the first mode, includes the following steps:

[0086] In response to determining that the image sensor is in a first mode, the photosensitivity of the image sensor is adjusted to a target value to reduce image noise during image acquisition; image acquisition is performed based on the adjusted photosensitivity to obtain the first image.

[0087] When the image sensor performs image acquisition in the first mode, the image acquisition noise in the binning mode can be reduced by changing the exposure mode and lowering the ISO (International Organization for Standardization, photosensitivity). For example, setting the target value to the minimum value helps reduce noise when the ISO is at the target value (minimum value).

[0088] This can further reduce the image noise level of the first image, which helps to obtain a more accurate first clustering result.

[0089] However, it's important to note that this method isn't universally applicable to all modes. Research has found that simply changing the exposure mode and adjusting ISO doesn't effectively reduce noise levels in remosaic mode. This is because the exposure mechanism of remosaic mode itself differs; it doesn't simply rely on a single combination of exposure time and ISO. In remosaic mode, image data undergoes special processing steps such as filtering and interpolation, resulting in different characteristics of the final processed data regardless of the exposure mode used. In other words, when remosaic uses exposure mode one, it undergoes an internal first processing step; when using exposure mode two, the internal first processing automatically adjusts to a second processing step and executes it. At this point, the true value of the data after the second processing step is not suitable as a noise-free true value pair with the data obtained using exposure mode one. In other words, under different exposure modes, data obtained by changing the exposure scheme cannot be directly considered as the true value unaffected by noise.

[0090] Optionally, the number of the second images is at least two, and no pixel position shift occurs between any two of the second images. The clustering accuracy of the first clustering result is higher than the clustering accuracy of clustering each pixel region of each second image.

[0091] To reduce the noise level of the second image, multiple second images can be acquired, and the average of the multiple second images can be calculated and merged into a single image.

[0092] For example, 10 frames of second images can be acquired in a second mode, and then the 10 frames of second images can be merged into 1 second image by calculating the position average. Here, it is required that no two second images will have pixel-level positional shifts, so as to ensure that the first image does not have pixel-level positional shifts compared with each second image (or with the merged second image).

[0093] In other words, when step 303 is executed in response to determining that the first image has not experienced a pixel position offset relative to the second image, it includes:

[0094] In response to determining that the first image has not produced a pixel position offset compared to either of the second images.

[0095] The method further includes the following steps:

[0096] In response to determining that the first image has a pixel position offset relative to either of the second images, the first image acquired by the image sensor in a first mode and the second image acquired in a second mode are reacquired.

[0097] Whether it is a pixel-level positional offset between any two second images or a pixel-level positional offset between the first image and any second image, it will result in a pixel-level positional offset between the first image and at least one second image. In this case, it is impossible to obtain a relatively accurate second clustering result based on the first clustering result, so it is necessary to reacquire the first image and the second image.

[0098] Optionally, the second mode is a remosaic mode; the number of the second images is less than or equal to a preset threshold, wherein the preset threshold is greater than or equal to 2.

[0099] In this disclosure, the following issues are considered:

[0100] When capturing multiple second images in remosaic mode, increasing the number of second images can mitigate some of the noise introduced by the interpolation algorithm in remosaic mode, but it cannot improve noise introduced by other processing algorithms within the image sensor. However, as the number of acquired second images increases to 40, 60, or even 100 frames, the acquisition device experiences severe overheating, resulting in significant thermal noise in the images. Therefore, when using remosaic mode as the second image, a reasonable acquisition number needs to be set to mitigate some noise while avoiding the generation of thermal noise.

[0101] For example, a preset threshold can be set to 10, 20, etc. In this case, 10 frames of the second image can be captured and fused to obtain a single second image.

[0102] Optionally, both the first and second images are raw images in their original image format. For example, if the first mode is binning mode and the second mode is remosaic mode, neither the first image captured in the first mode nor the second image captured in the second mode is a grayscale image. However, this scheme of first performing block matching (clustering) on ​​the image and then filtering for noise reduction is designed for grayscale images. Therefore, before performing clustering, the raw image needs to be converted into four grayscale images with different color channels. Then, clustering and filtering operations are performed on each grayscale image. Finally, the four grayscale images are combined into one raw image.

[0103] Therefore, the method further includes:

[0104] For each target image, a first format transformation is performed on the target image to obtain four grayscale images corresponding to different color channels; the target image includes the first image and the second image, and the first clustering result or the second clustering result includes: four clustering results for the four grayscale images respectively.

[0105] The first format transformation is the operation of converting one frame of raw image into four frames of grayscale images. The first clustering result is the result of clustering the four grayscale images separately, resulting in four clustering results corresponding to the four grayscale images. The clustering result of each grayscale image in the second clustering result is obtained based on the clustering results of grayscale images with the same color channel of the first image.

[0106] The four grayscale images correspond to the four color channels R, Gr, Gb, and B, respectively, and they represent the following meanings:

[0107] R (Red): The red channel records the intensity information of the red component in the image.

[0108] Gr (Green-Red or Green-R): Green-Red cross channel. In some advanced or special image sensors, there may be an additional green channel, which is more responsive to the red spectrum and less responsive to the green spectrum than the regular green channel (G). This design helps to improve the accuracy of color reproduction.

[0109] Gb (Green-Blue or Green-B): Green-blue cross channel, similar to Gr channel. In some image sensors, the Gb channel is also an additional green channel. It responds more strongly to the blue spectrum and less strongly to the green spectrum.

[0110] B (Blue): The blue channel records the intensity information of the blue component in the image.

[0111] At this point, the noise reduction of the second image based on the second clustering result includes:

[0112] Based on the clustering results corresponding to each grayscale image in the second clustering result, noise reduction is performed to obtain four grayscale images after noise reduction; a second format transformation is performed on the four grayscale images after noise reduction to obtain the original format raw image after noise reduction.

[0113] Similarly, the noise reduction process is performed separately for the four clustering results corresponding to the four grayscale images. Furthermore, after noise reduction, the four denoised grayscale images need to be restored to a single raw image.

[0114] Optionally, the clustering of each pixel region in the first image includes:

[0115] The first image is clustered by using a sliding window matching method to identify similar pixel regions.

[0116] In this disclosure, a sliding window matching method is used for clustering. For example, a sliding window search method can be used to find similar pixel regions using the L2 norm (or L1 norm) as an indicator to obtain clustering results.

[0117] It is important to note that when performing a sliding window search, an image is not directly divided into several non-overlapping pixel regions. Instead, a method of moving the window to traverse the image step by step is used. Each time, the window selects pixels within a certain neighborhood as an independent subset for cluster analysis. Therefore, after each window position is moved (i.e., the window is reset to the next position), there will be some overlapping areas that intersect with the previous window region. This ensures the continuity between adjacent regions and reduces boundary effects, thereby improving the integrity and stability of the clustering results.

[0118] The following explanation uses images as a basis, assuming the original intention was to... Figure 4a The pixel region 406 shown is acquired, and the acquired pixel region is finally as follows. Figure 5 As shown, in addition to acquiring 406, some pixels from adjacent pixel regions are also acquired; similarly, when acquiring 405, some pixels from 404, 406, and 402 are also acquired. This results in overlapping areas between the processed pixel regions when restoring them. Therefore, to address this issue, the process of restoring the denoised pixel regions to their original positions in the second image to obtain the denoised second image includes the following steps:

[0119] Each pixel region is reset to its original position in the second image to obtain an intermediate image; the average value of the overlapping pixels in the intermediate image is taken to obtain the noise-reduced second image.

[0120] In other words, for areas where pixels overlap, the average value can be calculated; for non-overlapping areas, the pixel values ​​of the denoised pixels can be directly taken.

[0121] In this way, we can obtain the second image after noise reduction.

[0122] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should know that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps may be performed in other orders or simultaneously.

[0123] Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by this disclosure.

[0124] Corresponding to the aforementioned application function implementation method embodiments, this disclosure also provides an embodiment of a noise reduction device and a corresponding terminal.

[0125] Figure 6 This is a schematic diagram of the structure of a noise reduction device according to an exemplary embodiment of the present disclosure, such as... Figure 6 As shown, the noise reduction device includes:

[0126] The first acquisition module 601 is used to acquire a first image acquired by the image sensor in a first mode and a second image acquired in a second mode.

[0127] The first clustering module 602 is used to cluster each pixel region in the first image to obtain a first clustering result; the clustering accuracy of the first clustering result is higher than that of the clustering result of clustering each pixel region of the second image.

[0128] The second acquisition module 603 is used to acquire the correspondence between each pixel in the first image and the second image in response to determining that the first image has not produced a pixel position offset compared to the second image.

[0129] The second clustering module 604 is used to obtain a second clustering result for each pixel region in the second image based on the correspondence and the first clustering result.

[0130] The noise reduction module 605 is used to reduce noise in the second image based on the second clustering result.

[0131] Optionally, the resolution of the first image is different from the resolution of the second image.

[0132] The second acquisition module 603, in response to determining that the first image has not undergone a pixel position offset relative to the second image, is configured to:

[0133] The resolution of the first image is set to be the same as the resolution of the second image.

[0134] It was determined that the first image and the second image did not produce pixel position offset at the same resolution.

[0135] Optionally, when the noise reduction module 605 performs noise reduction on the second image based on the second clustering result, it is used to:

[0136] Noise reduction is performed on the pixel regions in each cluster of the second clustering result.

[0137] The denoised pixel regions are then restored to their original positions in the second image to obtain the denoised second image.

[0138] Optionally, the first mode is the binning mode of the image sensor.

[0139] When the first acquisition module 601 acquires the first image captured by the image sensor in a first mode, it is used to:

[0140] In response to determining that the image sensor is in a first mode, the photosensitivity of the image sensor is adjusted to a target value to reduce image noise during image acquisition.

[0141] The first image is obtained by acquiring an image based on the adjusted photosensitivity.

[0142] Optionally, the number of the second images is at least two, and no pixel position shift occurs between any two of the second images. The clustering accuracy of the first clustering result is higher than the clustering accuracy of clustering each pixel region of each second image.

[0143] Optionally, the second mode is a remosaic mode; the number of the second images is less than or equal to a preset threshold, wherein the preset threshold is greater than or equal to 2.

[0144] At this time, the second acquisition module 603, in response to determining that the first image has not produced a pixel position offset relative to the second image, is used to:

[0145] In response to determining that the first image has not produced a pixel position offset compared to either of the second images.

[0146] The device further includes:

[0147] The reacquisition module is configured to, in response to determining that the first image has a pixel position offset relative to either of the second images, reacquire the first image acquired by the image sensor in a first mode and the second image acquired in a second mode.

[0148] Optionally, both the first image and the second image are raw images in their original image format.

[0149] The device further includes:

[0150] The first transformation module is used to perform a first format transformation on each target image to obtain four grayscale images corresponding to different color channels; the target image includes the first image and the second image, and the first clustering result or the second clustering result includes four clustering results for the four grayscale images respectively.

[0151] When the noise reduction module 605 is used to reduce noise in the second image based on the second clustering result, it is used to:

[0152] Denoising is performed on the clustering results corresponding to each grayscale image in the second clustering result to obtain four denoised grayscale images. A second format transformation is then performed on the four denoised grayscale images to obtain the denoised original format raw image.

[0153] Optionally, when the first clustering module 602 clusters the pixel regions in the first image, it is used to:

[0154] The first image is clustered by using a sliding window matching method to identify similar pixel regions.

[0155] The noise reduction module 605 is used to reset the noise-reduced pixel region to its original position in the second image to obtain the noise-reduced second image, for the following purposes:

[0156] Each pixel region is reset to its original position in the second image to obtain an intermediate image; the average value of the overlapping pixels in the intermediate image is taken to obtain the noise-reduced second image.

[0157] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0158] Accordingly, this disclosure provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of any of the above noise reduction methods by running the executable instructions.

[0159] Figure 7 This disclosure is a schematic diagram illustrating the structure of an electronic device according to an exemplary embodiment. For example, the electronic device 700 can be a user device, specifically a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, wearable device such as smartwatch, smart glasses, smart bracelet, smart running shoes, etc.

[0160] Reference Figure 7 The electronic device 700 may include one or more of the following components: a processing component 702, a memory 704, a power supply component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0161] Processing component 702 typically controls the overall operation of electronic device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0162] Memory 704 is configured to store various types of data to support the operation of device 700. Examples of this data include instructions for any application or method operating on electronic device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0163] Power supply component 706 provides power to various components of electronic device 700. Power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 700.

[0164] Multimedia component 708 includes a screen that provides an output interface between the aforementioned electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0165] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when electronic device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0166] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0167] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of electronic device 700. For example, sensor assembly 714 can detect the on / off state of electronic device 700, the relative positioning of components such as the display and keypad of electronic device 700, changes in position of electronic device 700 or one of its components, the presence or absence of user contact with electronic device 700, orientation or acceleration / deceleration of electronic device 700, and temperature changes of electronic device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0168] Communication component 716 is configured to facilitate wired or wireless communication between electronic device 700 and other devices. Electronic device 700 can access wireless networks based on communication standards, such as WiFi, 4G or 5G, 4G LTE, 5G NR, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the aforementioned communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0169] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0170] In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided, such as a memory 704 including instructions, which, when executed by a processor 720 of an electronic device 700, enables the electronic device 700 to perform any of the noise reduction methods described above.

[0171] The non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0172] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0173] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A noise reduction method, characterized in that, The method includes: Acquire a first image captured by an image sensor in a first mode and a second image captured in a second mode; Clustering is performed on each pixel region in the first image to obtain a first clustering result; the clustering accuracy of the first clustering result is higher than the clustering accuracy of clustering each pixel region in the second image. In response to determining that the first image has no pixel position offset compared to the second image, the correspondence between each pixel in the first image and the second image is obtained; Based on the correspondence and the first clustering result, a second clustering result is obtained for each pixel region in the second image; The second image is denoised based on the second clustering result.

2. The method according to claim 1, characterized in that, The resolution of the first image is different from the resolution of the second image; The response to determining that the first image has not produced a pixel position offset compared to the second image includes: Set the resolution of the first image to be the same as the resolution of the second image; It was determined that the first image and the second image did not produce pixel position offset at the same resolution.

3. The method according to claim 1, characterized in that, The noise reduction of the second image based on the second clustering result includes: Denoise reduction is performed on the pixel regions in each cluster of the second clustering result; The denoised pixel regions are then restored to their original positions in the second image to obtain the denoised second image.

4. The method according to claim 1, characterized in that, The first mode is the binning mode of the image sensor; The acquisition of the first image captured by the image sensor in a first mode includes: In response to determining that the image sensor is in a first mode, the photosensitivity of the image sensor is adjusted to a target value to reduce image noise during image acquisition; The first image is obtained by acquiring an image based on the adjusted photosensitivity.

5. The method according to claim 1, characterized in that, The number of the second images is at least two, and no pixel position shift occurs between any two of the second images. The clustering accuracy of the first clustering result is higher than the clustering accuracy of clustering each pixel region of each second image.

6. The method according to claim 5, characterized in that, The second mode is remosaic mode; The number of the second image is less than or equal to a preset threshold, and the preset threshold is greater than or equal to 2.

7. The method according to claim 5, characterized in that, The response to determining that the first image has not produced a pixel position offset compared to the second image includes: In response to determining that the first image has not produced a pixel position offset relative to either of the second images; The method further includes: In response to determining that the first image has a pixel position offset relative to either of the second images, the first image acquired by the image sensor in a first mode and the second image acquired in a second mode are reacquired.

8. The method according to claim 1, characterized in that, Both the first image and the second image are raw images in original image format; The method further includes: For each target image, a first format transformation is performed on the target image to obtain four grayscale images corresponding to different color channels; the target image includes the first image and the second image, and the first clustering result or the second clustering result includes: four clustering results for the four grayscale images respectively; The noise reduction of the second image based on the second clustering result includes: Denoising is performed based on the clustering results corresponding to each grayscale image in the second clustering result to obtain four grayscale images after denoising. A second format transformation is performed on the four denoised grayscale images to obtain the original format raw image after denoising.

9. The method according to claim 3, characterized in that, The clustering of each pixel region in the first image includes: Clustering is performed on similar pixel regions in the first image using a sliding window matching method; The step of restoring the denoised pixel region to its original position in the second image to obtain the denoised second image includes: Each pixel region is reset to its original position in the second image to obtain the intermediate image; The average value of overlapping pixels in the intermediate image is taken to obtain the second image after noise reduction.

10. A noise reduction device, characterized in that, The device includes: The first acquisition module is used to acquire a first image acquired by the image sensor in a first mode and a second image acquired in a second mode; The first clustering module is used to cluster each pixel region in the first image to obtain a first clustering result; the clustering accuracy of the first clustering result is higher than that of the clustering result of clustering each pixel region of the second image. The second acquisition module is used to acquire the correspondence between each pixel in the first image and the second image in response to determining that the first image has not produced a pixel position offset relative to the second image; The second clustering module is used to obtain a second clustering result for each pixel region in the second image based on the correspondence and the first clustering result; A noise reduction module is used to reduce noise in the second image based on the second clustering result.

11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of the method according to any one of claims 1 to 9 by running the executable instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 9.

Images