Image denoising method, electronic device, and storage medium

Abstract

The application discloses an image denoising method, an electronic device and a storage medium, relates to the technical field of image processing, and can reduce the influence on the picture quality while denoising. Specifically, for an image to be denoised, an electronic device can obtain more accurate noise intensity information 2 corresponding to each pixel point in the image. Since the noise intensity information 2 is determined by the denoising strength corresponding to the pixel point and the noise intensity information 1 estimated in advance for the pixel point, the noise intensity information 2 is more accurate than the noise intensity information 1. Furthermore, the electronic device can input the noise intensity information 2 as additional supplementary prior information to a denoising network together with the first image, so as to more accurately guide the denoising network to perform accurate denoising, thereby reducing the influence of the picture quality reduction caused by denoising.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image denoising method, electronic device, and storage medium. Background Technology

[0002] With the rapid development of image processing technology, image denoising and image quality enhancement have become a research hotspot in the field of image processing. How to balance noise and sharpness is a major challenge for the industry.

[0003] Currently, most industry practices use neural networks for image denoising. This involves inputting the image to be denoised into a neural network with denoising capabilities, and then outputting the denoised image. However, neural networks exhibit significant instability in image denoising, often resulting in a substantial degradation in image quality.

[0004] Therefore, it is necessary to propose a solution that can reduce the impact on image quality while denoising. Summary of the Invention

[0005] This application provides an image denoising method, electronic device, and storage medium that can reduce the impact on image quality while denoising.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] Firstly, this application provides an image denoising method applied to an electronic device. For a first image to be denoised, the electronic device can obtain more accurate noise intensity information 2 corresponding to each pixel in the first image; that is, the noise intensity information 2 is more accurate than the noise intensity information 1 obtained by estimating the noise intensity of the first image. Furthermore, the electronic device can input the noise intensity information 2 as supplementary prior information along with the first image into a denoising network, thereby more accurately guiding the denoising network to perform precise denoising, mitigating the image quality degradation caused by denoising, and obtaining an image 2 with better image quality after denoising.

[0008] The reason why noise intensity information 2 is more accurate than noise intensity information 1 obtained by noise intensity estimation is that:

[0009] The noise intensity information 2 corresponding to each pixel is determined by the denoising intensity corresponding to that pixel and the estimated noise intensity information 1 for that pixel. The denoising intensity of pixels in the first region of the first image is less than that of pixels in the second region of the first image; the first region has more texture information than the second region. That is, the denoising intensity of regions with more texture information in the first image is less than that of regions with less texture information. Therefore, based on the estimated noise intensity information 1, the noise intensity information of pixels in regions with more texture information can be relatively reduced, thus mitigating the problem of excessive denoising in regions with more texture, leading to over-smearing. Similarly, based on the estimated noise intensity information 1, the noise intensity information of pixels in regions with less texture information can be relatively increased, thereby increasing the denoising effect of the denoising network on regions with more texture, avoiding or mitigating the problem of pseudo-textures due to inadequate denoising. Therefore, the above scheme can reduce the impact on image quality when the denoising network denoises the image, outputting an image with better denoised quality, thereby improving the user experience.

[0010] In one possible implementation of the first aspect, the electronic device itself can estimate the noise intensity of the first image to obtain the noise intensity information 1 corresponding to each pixel in the first image. Then, based on the denoising effort corresponding to each pixel, the estimated noise intensity information 1 of the pixel is adjusted to obtain the noise intensity information 2 corresponding to the pixel.

[0011] In the above scheme, the noise intensity is estimated by the electronic device itself, without the need to interact with other devices to obtain the estimated noise intensity information. This reduces the problem of untimely processing caused by external interaction, and enables the entire image denoising process to be completed on the electronic device itself. This improves the efficiency of image denoising and is more suitable for real-time scenarios such as taking pictures and recording videos, making it more widely applicable.

[0012] In one possible implementation of the first aspect, the electronic device can determine the denoising intensity corresponding to each pixel based on the quantization value corresponding to each pixel in the first image. The quantization value characterizes the amount of texture information corresponding to the pixel, and the magnitude of the quantization value corresponding to each pixel is positively correlated with the amount of texture information corresponding to the pixel, while the magnitude of the quantization value corresponding to each pixel is negatively correlated with the magnitude of the denoising intensity.

[0013] In the above scheme, the texture information is quantized to obtain the quantization value corresponding to each pixel, thereby determining the denoising intensity of that pixel. This achieves pixel-level denoising control, greatly improving the precision of denoising control and thus enhancing the denoising effect.

[0014] In one possible implementation of the first aspect, each pixel corresponds to the quantization value, including the pixel value variance or the gradient value corresponding to the pixel; wherein, the pixel value variance corresponding to each pixel is the variance of the pixel values ​​of multiple pixels in the neighborhood of the pixel.

[0015] In the above scheme, the pixel value variance or the gradient value corresponding to the pixel can quantify the amount of texture information (or texture information quantity) corresponding to the pixel, so as to accurately distinguish the regions with different texture information in the first image, and then more accurately determine the denoising intensity corresponding to the pixel in the subsequent process, thereby improving the accuracy of pixel-level denoising control.

[0016] In one possible implementation of the first aspect, the electronic device can perform semantic segmentation on the third image to obtain multiple sub-regions; the third image is the first image or an image after smoothing the first image; for each pixel in each sub-region, the denoising intensity corresponding to the pixel is determined according to the variance of the pixel value corresponding to the pixel.

[0017] In the above scheme, semantic segmentation and pixel variance control are combined to determine the denoising intensity of pixels in different regions by comprehensively considering both texture information differences and semantic information differences, which greatly improves the accuracy of denoising intensity control.

[0018] In addition, image smoothing can initially identify some interference noise, and variance calculation after smoothing can more accurately determine the denoising level.

[0019] In one possible implementation of the first aspect, after semantic segmentation, for each sub-region, the electronic device pre-sets a correspondence between the region type and a preset variance threshold. Based on the region type, a variance threshold corresponding to the region type of each semantically segmented sub-region is determined. For each pixel in the sub-region, the variance of the pixel value corresponding to the pixel is compared with the determined variance threshold, and the denoising intensity corresponding to the pixel is determined based on the comparison result.

[0020] In the above scheme, corresponding variance thresholds are configured based on different region types, and specific denoising intensity is configured for each variance threshold in the variance threshold set, which can more accurately determine the denoising intensity of pixels in each region. This, in turn, improves the accuracy of subsequent denoising. Furthermore, this approach is based on the flexible configuration of the corresponding variance threshold set and denoising intensity for different region types, offering flexibility and adjustability suitable for any scenario.

[0021] In one possible implementation of the first aspect, the electronic device can determine the denoising intensity of a pixel based on the quantization value and the pixel attribute value of that pixel.

[0022] The above scheme can consider more dimensions of information when determining the denoising intensity, so it can not only differentiate between areas with less texture and areas with more texture, but also differentiate between areas with different pixel attribute values. This makes the denoised image more accurately represent the actual environment or better meet the actual needs, thus improving the accuracy of the denoising intensity.

[0023] In one possible implementation of the first aspect, the pixel attribute value can be a brightness value. The electronic device can determine the denoising intensity corresponding to a pixel based on the quantization value and the brightness value of that pixel. Specifically, for each pixel in the first image, a first sub-denoising intensity is determined based on the quantization value corresponding to the pixel; the quantization value is used to characterize the amount of texture information corresponding to the pixel; the magnitude of the quantization value corresponding to each pixel is positively correlated with the amount of texture information corresponding to the pixel; a second sub-denoising intensity is determined based on the brightness value corresponding to the pixel; and the denoising intensity corresponding to the pixel is determined based on the first sub-denoising intensity and the second sub-denoising intensity corresponding to the pixel.

[0024] Since brightness values ​​are strongly correlated with image denoising requirements, and bright and dark areas have a significant impact on user vision, the above solution uses the electronic device to determine the denoising intensity of each pixel based on its quantization value and brightness value. This improves denoising accuracy and makes the denoised image more in line with user visual needs.

[0025] In one possible implementation of the first aspect, the noise intensity information 2 corresponding to each pixel is represented by a noise intensity map; the electronic device can superimpose the noise intensity map and the first image to obtain a fourth image corresponding to the first image; the fourth image is input to the denoising network for denoising processing to obtain the second image.

[0026] In the above scheme, by superimposing a noise intensity map, which represents more accurate noise intensity information, onto the first image (e.g., channel superposition), a new image is generated. This new image contains more prior information, thus accurately guiding the denoising network to perform precise denoising. Furthermore, generating a new image through channel superposition, rather than mixing and fusing information, allows for more accurate guidance of the denoising network, further improving denoising accuracy.

[0027] In one possible implementation of the first aspect, there are multiple first images; the multiple first images are temporally adjacent; each first image has a corresponding noise intensity map. An electronic device can superimpose each first image with its corresponding noise intensity map to obtain a fifth image corresponding to each first image; superimpose the fifth images corresponding to the multiple first images respectively to obtain a sixth image; and input the sixth image into the denoising network for denoising processing to obtain the second image.

[0028] In the above scheme, multiple temporally adjacent images are input into the denoising network, which increases the amount of input information. This allows the denoising network to refer to more information for denoising, resulting in a clearer and higher-quality denoised image that is output by the denoising network. This achieves a good balance between image quality and clarity.

[0029] In one possible implementation of the first aspect, after the denoising network outputs a denoised second image (i.e., a denoised image), the electronic device can further perform detail enhancement processing on the denoised image. Specifically, the electronic device can acquire detail feature information; the detail feature information is used to characterize the detail features in the first image; and the detail feature information is fused into the second image to obtain a seventh image.

[0030] In the above scheme, after obtaining the denoised image output by the denoising network, the texture details in the denoised image can be improved through detail enhancement processing, thereby further reducing the impact of denoising on image quality.

[0031] In one possible implementation of the first aspect, the detailed feature information is represented by a detailed image. The electronic device can split the detailed image into a first low-frequency detailed image and a first high-frequency detailed image; each pixel in the second image corresponds to a set of superposition weights, the set of superposition weights including a first superposition weight and a second superposition weight, the magnitude of the first superposition weight and the second superposition weight in the set of superposition weights corresponding to each pixel being determined by the magnitude of the brightness value corresponding to the pixel in the second image; for each pixel in the second image, the pixel value corresponding to the pixel in the first low-frequency detailed image is superimposed according to the first superposition weight, and the pixel value corresponding to the pixel in the first high-frequency detailed image is superimposed according to the second superposition weight, with the pixel value corresponding to the pixel in the second image, to obtain the seventh image.

[0032] In the above scheme, after splitting the detail image into a first low-frequency detail image and a first high-frequency detail image, the superposition weight of the low-frequency detail and the superposition weight of the high-frequency detail corresponding to the pixel are determined according to the brightness value of the pixel. Then, the low-frequency detail and the high-frequency detail are superimposed and fused into the noise-reduced image according to the corresponding superposition weights. This achieves fine superposition control of low-frequency detail and high-frequency detail at the pixel level, which greatly improves the accuracy of detail enhancement.

[0033] In one possible implementation of the first aspect, the electronic device can multiply the pixel value corresponding to each pixel in the first low-frequency detail image with the first superposition weight corresponding to the pixel to obtain a second low-frequency detail image; multiply the pixel value corresponding to each pixel in the first high-frequency detail image with the second superposition weight corresponding to the pixel to obtain a second high-frequency detail image; and superimpose the second low-frequency detail image and the second high-frequency detail image onto the second image to obtain the seventh image.

[0034] In the above scheme, the pixel values ​​corresponding to each pixel in the low-frequency detail image and the high-frequency detail image are multiplied by the corresponding superposition weights to make adjustments, generating new low-frequency detail images and high-frequency detail images. Then, they are superimposed on the denoised image. At the data processing level, this is both efficient and accurate.

[0035] In one possible implementation of the first aspect, the electronic device can convert the second image to the YUV domain to extract luminance channel information to obtain a luminance map; for each pixel in the second image, a first superposition weight and a second superposition weight corresponding to the pixel are determined according to the luminance value corresponding to the pixel in the luminance map; wherein, the first superposition weight corresponding to each pixel is used to characterize the superposition degree of the pixel value corresponding to the first low-frequency detail map; and the second superposition weight corresponding to each pixel is used to characterize the superposition degree of the pixel value corresponding to the first high-frequency detail map.

[0036] In the above scheme, based on the brightness value corresponding to each pixel in the denoised image, the superposition weight of high-frequency details and low-frequency details corresponding to the pixel can be determined more accurately, which is more accurate.

[0037] Secondly, this application provides an electronic device comprising at least a display screen, a memory, and one or more processors. The display screen is used to display images, and the memory is used to store computer instructions. When the one or more processors execute the computer instructions, the electronic device performs the method described in any of the first aspects above.

[0038] Thirdly, this application provides a chip system applied to an electronic device, the chip system including one or more processors, the processors being used to invoke computer instructions to cause the electronic device to perform any of the methods described in the first aspect above.

[0039] Fourthly, this application provides a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform any of the methods described in the first aspect above.

[0040] Fifthly, this application provides a computer program product that, when run on a computer, causes the computer to perform any of the methods described in the first aspect above. Attached Figure Description

[0041] Figure 1 A schematic diagram illustrating the inadequate noise reduction using conventional methods provided in the embodiments of this application;

[0042] Figure 2 A schematic diagram of the principle of the image denoising method provided in the embodiments of this application. Figure 1 ;

[0043] Figure 3 A schematic diagram of the principle of the image denoising method provided in the embodiments of this application. Figure 2 ;

[0044] Figure 4 A schematic diagram of a scene for the image denoising method provided in an embodiment of this application;

[0045] Figure 5 A hardware structure block diagram of an electronic device provided in an embodiment of this application;

[0046] Figure 6 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0047] Figure 7 A schematic diagram illustrating a scenario in which the image denoising method provided in this application embodiment is applied;

[0048] Figure 8A This is a semantic segmentation diagram provided for an embodiment of this application;

[0049] Figure 8B A schematic diagram illustrating the calculation of pixel value variance provided in an embodiment of this application;

[0050] Figure 8C A schematic diagram illustrating the principle of determining the denoising intensity provided in an embodiment of this application;

[0051] Figure 9 A schematic diagram of the process for obtaining prior noise intensity information provided in an embodiment of this application;

[0052] Figure 10 A schematic diagram illustrating the principle of single-image overlay provided in an embodiment of this application;

[0053] Figure 11 A schematic diagram illustrating the principle of multiple image overlay provided in the embodiments of this application;

[0054] Figure 12 Schematic diagram of the principle of detail enhancement processing provided in the embodiments of this application Figure 1 ;

[0055] Figure 13 A schematic diagram of the principle of the image denoising method provided in the embodiments of this application. Figure 3 ;

[0056] Figure 14 Schematic diagram of the principle of detail enhancement processing provided in the embodiments of this application Figure 2 . Detailed Implementation

[0057] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In the description of the embodiments, unless otherwise stated, "multiple" means two or more.

[0058] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. The terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature.

[0059] In the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0060] The solution disclosed in this application is mainly applied to image denoising scenarios, aiming to solve the problem of significant image quality degradation caused by using denoising networks for image denoising. In other words, the solution disclosed in this application can reduce the impact on image quality when using denoising networks for image denoising.

[0061] Before formally introducing the solution of this application, the concepts related to the solution of this application will be introduced as follows:

[0062] Noise (image noise): refers to noise information in an image, that is, unnecessary and redundant interference information in the image.

[0063] Noise intensity (noise intensity information): This refers to the strength of noise information present in an image, representing the significance or energy of the noise in the image. In other words, noise intensity is an indicator of the degree to which noise affects the image, reflecting the extent to which noise interferes with image quality. Higher noise intensity indicates that the noise is more significant in the image and interferes more with image quality; conversely, lower noise intensity indicates that the noise has a smaller impact on the image, and the image quality is relatively better.

[0064] Noise intensity estimation refers to the process of estimating the intensity of noise in an image. Generally, a mathematical model is used to model the noise in the image, and then the noise intensity is estimated based on this model.

[0065] Noise models may include Gaussian noise models, Poisson-Gaussian noise models, or other noise models with different structures. The embodiments of this application do not limit the structure of the noise model.

[0066] It should be understood that constructing a noise model involves fitting a mathematical model to the noise in an image, essentially using a fixed formula. In this noise model, the relationship between noise magnitude and pixel values ​​in the image is fixed. For a given pixel, a larger pixel value corresponds to greater noise in the noise model. Therefore, the noise intensity estimated using the noise model will inevitably be greater, leading to stronger denoising for pixels with larger values ​​based on the estimated noise intensity. This method of noise intensity estimation is too absolute, often resulting in over-denoising of areas with abundant texture information and large pixel values. Similarly, it can lead to insufficient denoising for areas with less texture information and fewer pixel values, resulting in pseudo-textures.

[0067] Denoising network: This is a pre-trained (deep) neural network used for image denoising. The denoising network can be trained using network structures such as U-Net, ResNet, and Transformer. The network structure of the denoising network is not limited in the various embodiments of this application.

[0068] The proposal will now be formally introduced.

[0069] In scenarios involving image denoising, neural networks with denoising capabilities (hereinafter referred to as denoising networks) are generally used to denoise images. However, when using denoising networks to denoise images, due to the instability of the denoising networks, the identification of noise information in the image is not accurate enough. As a result, the denoised image output by the denoising network is prone to some areas (such as textured areas) being overly denoised, while some areas (such as flat areas) have pseudo-textures.

[0070] like Figure 1 As shown, Figure 1 (a) is used to illustrate the case of excessive denoising in the texture area. Figure 1 In the image shown in (a), region 101 should ideally have relatively clear texture information. However, after denoising processing by a denoising network, it appears as follows: Figure 1The effect shown in (a) is that the denoising is too heavy, resulting in the loss of some textures and the appearance of a patchy area 101, which leads to a decrease in image quality. Figure 1 (b) is used to illustrate the situation where poor noise reduction in flat areas produces pseudo-textures. Figure 1 In the image shown in (b), region 102 should ideally be flat, with little or no texture. However, after denoising by a denoising network, it appears as follows: Figure 1 The effect shown in (b) is that the denoising is not balanced and stable, which leads to the appearance of false textures in region 102, resulting in a decrease in image quality.

[0071] In some solutions, to achieve more accurate denoising and reduce the impact of improper denoising on image quality, a noise intensity estimation algorithm is used to estimate the noise intensity of the image to be denoised, predicting / estimating the noise intensity information present in the image. This noise intensity information characterizes the strength of the noise present in the image. Then, the image along with the estimated noise intensity information is input into the denoising network. The noise intensity information guides the denoising network to a certain extent, allowing it to refer to more information during denoising processing, thereby improving denoising accuracy and reducing the impact of denoising on image quality.

[0072] However, the inventors of this application discovered that, under normal circumstances, the noise intensity information obtained from noise intensity estimation is not accurate enough, resulting in limited improvement in image quality during denoising scenarios. Furthermore, through in-depth research, the inventors found that the reason for the inaccuracy of the estimated noise intensity information is that, in the noise model constructed using a mathematical model, the noise level of each pixel in the image is positively correlated with the pixel value. Consequently, when performing noise intensity estimation based on the noise model, the estimated noise intensity information is also positively correlated with the pixel value. That is, the larger the pixel value of a pixel, the greater the estimated noise intensity information for that pixel, and vice versa.

[0073] However, in reality, the noise intensity of a pixel in an image is not necessarily positively correlated with its pixel value. For example, regions with more texture in an image do not necessarily have higher noise intensity, and regions with less texture do not necessarily have lower noise intensity. If a denoising network performs denoising based on estimated noise intensity information, it will apply greater denoising force to regions with more texture, leading to over-denoising and other problems. Figure 1 The example in (a) shows an overly heavy application of smearing. Similarly, the denoising network may not be strong enough in areas with less texture, resulting in... Figure 1 The pseudo texture shown in (b).

[0074] To address the aforementioned issues, this application proposes an image denoising method in some embodiments, applicable to electronic devices. In scenarios involving image denoising, this method can acquire more accurate noise intensity information in the image to be denoised / the noisy image (i.e., image 1 or the first image). This more accurate noise intensity information is used as additional compensation information and input together with the image to be denoised into a denoising network. This allows the denoising network to combine the more accurate noise intensity information (i.e., prior noise intensity information) to perform precise denoising on the image, thereby mitigating the impact of image quality degradation caused by denoising. After denoising, an image with better image quality (i.e., image 2 or the second image) is obtained.

[0075] It should be understood that in the implementation of this application, the prior noise intensity information is more accurate than the noise intensity information estimated by the noise intensity estimation. That is, it can more accurately measure and reflect the noise intensity of each region or pixel in the noisy image (the image to be denoised), thereby playing a priori role and more accurately guiding the denoising network to denoise.

[0076] For details, please refer to [link / reference]. Figure 2 ,like Figure 2 As shown, the noisy image is input to the prior information extraction module to obtain a prior noise intensity map (used to characterize prior noise intensity information). Then, the prior noise intensity map and the noisy image can be superimposed to form a new image. The superimposed new image is then input into the denoising network to obtain a denoised image with better image quality.

[0077] It should be noted that the more accurate noise intensity information obtained above is not derived from noise intensity estimation. The electronic device combines the noise intensity information obtained from the noise intensity estimation (i.e., noise intensity information 1, or first noise intensity information) with the denoising intensity corresponding to each pixel in the image to jointly determine the more accurate noise intensity information (i.e., noise intensity information 2, or second noise intensity information). The magnitude of the denoising intensity corresponding to each pixel is negatively correlated with the amount of texture information corresponding to that pixel or the amount of texture information in the region where that pixel is located. For example, for the first region and the second region, the denoising intensity corresponding to pixels in the first region, which has relatively more texture information, is less than the denoising intensity corresponding to pixels in the second region, which has relatively less texture information.

[0078] For details, please refer to [link / reference]. Figure 3 ① The prior information extraction module can estimate the noise intensity of noisy images to obtain the estimated noise intensity. Figure 1 ② The prior information extraction module can determine the denoising intensity corresponding to each pixel in the noisy image. ③ The prior information extraction module can adjust the estimated noise intensity according to the denoising intensity. Figure 1④ After adjustment, the prior noise intensity can be obtained. Figure 2 ⑤ Combine the noisy image with the prior noise intensity. Figure 2 ⑥ The superimposed images are input into a denoising network to obtain a noisy output image.

[0079] This approach avoids or reduces the possibility of over-denoising areas with high texture based on estimated noise intensity information, preventing excessive denoising in these areas and preserving appropriate texture details. Similarly, it avoids or reduces insufficient denoising in areas with low texture, removing unwanted textures. In short, by applying different denoising intensities to areas with varying texture information, it better balances noise and sharpness, reducing the impact of denoising on image quality and improving overall image quality.

[0080] For example, the prior information extraction module is an adjustable module. In the following description, when generating the prior noise intensity information 2, the denoising intensity is determined, and this intensity is determined based on pre-configured parameter relationships. Therefore, the denoising intensity can be controlled by changing the configuration, thereby adjusting the prior noise intensity information 2 input to the denoising network. This demonstrates the adjustable nature of the prior information extraction module. By flexibly changing the input to the denoising network, the denoising effect can be accurately controlled according to requirements.

[0081] The following section introduces specific scenarios involving image denoising.

[0082] It should be understood that the image denoising method in the embodiments of this application can be applied to a variety of scenarios, including but not limited to photography scenarios, video recording scenarios, and security scenarios, and any scenario that requires image acquisition. Using the method of this application can improve the image quality of the acquired images.

[0083] Scene 1: Taking photos.

[0084] Please see Figure 4 Suppose you need to take a picture of a cabinet in a room. The phone displays a camera interface, and after the user triggers control 201 on the camera interface, the phone's camera can capture an image. If, using traditional methods, only a denoising network is used for denoising, the final photo will be like image 20, with poor image quality. For example, area 202 in the cabinet is a textured area; in image 20, area 202 will show mottled areas because improper denoising results in excessive smoothing. The phone can execute the image denoising methods mentioned in the embodiments of this application to denoise the original captured image, obtaining a result like... Figure 4 Image 10 shows that the areas with more texture in the cabinet are painted more evenly, and the texture details are preserved more clearly, resulting in better image quality.

[0085] Scenario 2: Video recording scenario.

[0086] It should be understood that the image denoising methods in the embodiments of this application are not only applicable to photography scenarios, but also to video recording scenarios. In video recording scenarios, by executing this solution, the mobile phone can perform more accurate denoising processing on each captured video frame, resulting in better image quality for each denoised video frame, thereby forming a higher quality video.

[0087] Scenario 3: Security Scenario.

[0088] It should be noted that the security scenarios mentioned here can be home security scenarios (e.g., users installing cameras in their own homes for self-protection) or legal, authorized security scenarios. Taking home security scenarios as an example, after a user installs a camera at home, the camera can capture images of the home environment in real time. The camera can be a smart camera or a regular camera. If it is a smart camera, then the smart camera itself can embed the image denoising method processing logic of this application embodiment, thereby accurately denoising the real-time captured images. If it is a regular camera, then the camera can report the captured images in real time to a device capable of executing the logic of this solution, enabling that device to accurately denoise the captured images.

[0089] It should be noted that applying this solution to security scenarios can provide higher quality and clearer images, making the images a better reference when providing security protection, and improving the security protection effect to a certain extent.

[0090] For example, the aforementioned electronic devices may specifically include mobile phones, tablets, in-vehicle devices, security devices, dashcams, televisions (also known as smart TVs, smart screens, or large-screen devices), laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), handheld computers, wearable electronic devices (e.g., smartwatches, smart bracelets, smart glasses), virtual reality devices, and other terminal devices, or servers. This application does not impose any special limitations on the specific form of the electronic devices.

[0091] It should be understood that if the electronic device is a terminal device, then if the terminal device has image acquisition capabilities, it can acquire images and perform image denoising processing itself. If the electronic device is a server, then the server can acquire the images acquired by the terminal device and denoise them. Furthermore, the server can return the denoised image to the terminal device for display.

[0092] The following text mainly uses mobile phones as an example to illustrate the hardware and software structure of electronic devices.

[0093] See Figure 5 See Figure 5 The mobile phone may include a processor 510, an external memory interface 520, an internal memory 521, a universal serial bus (USB) interface 530, a charging management module 540, a power management module 541, a battery 542, antenna 1, antenna 2, a mobile communication module 550, a wireless communication module 560, an audio module 570, a speaker 570A, a receiver 570B, a microphone 570C, a headphone jack 570D, a sensor 580, buttons 590, a motor 591, an indicator 592, a camera 593, a display screen 594, and a user identification module (SIM) card interface 595, etc.

[0094] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the mobile phone. In other embodiments of this application, the mobile phone may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0095] Processor 510 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0096] In some embodiments, the mobile phone can perform image denoising by the processor 510, thereby reducing the impact on image quality during denoising.

[0097] The wireless communication function of a mobile phone can be implemented through antenna 1, antenna 2, mobile communication module 550, wireless communication module 560, modem processor, and baseband processor.

[0098] The mobile phone implements its display function through a GPU, a display screen 594, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 594 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 510 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0099] The display screen 594 is used to display images, videos, etc. In some embodiments, the mobile phone can display denoised images (i.e., denoised images) through the display screen 594.

[0100] The mobile phone can achieve shooting functions through I SP, camera 593, video codec, GPU, display 594 and application processor.

[0101] The mobile phone can perform audio functions, such as music playback and recording, through the audio module 570, speaker 570A, receiver 570B, microphone 570C, headphone jack 570D, and application processor.

[0102] Buttons 590 may include a power button, volume buttons, etc. Buttons 590 can be mechanical buttons or touch buttons. The phone can receive button input and generate key signal inputs related to user settings and function control. Motor 591 can generate vibration alerts. Motor 591 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 592 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. SIM card interface 595 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 595 to achieve contact and separation with the phone.

[0103] Mobile phone software systems can adopt layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses a layered architecture software system as an example to illustrate the software structure of a mobile phone. It should be noted that this application is not limited to the software structure of mobile phones.

[0104] Please see Figure 6 , Figure 6 This is a software structure block diagram of a mobile phone provided in an embodiment of this application.

[0105] As we can understand, a layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. For example... Figure 6 As shown, a software system may include an application layer, an application framework layer, and a hardware abstraction layer (HAL).

[0106] The application layer can include a series of application packages, which may include applications such as camera and gallery.

[0107] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions.

[0108] In some embodiments, the application framework layer may include a camera access interface, which may include interfaces for camera management and camera device access. The camera access interface provides an application programming interface and programming framework for camera applications.

[0109] The hardware abstraction layer is an interface layer located between the application framework layer and the kernel layer, providing a virtual hardware platform for the operating system.

[0110] In this embodiment, the hardware abstraction layer may include a camera hardware abstraction layer. The camera hardware abstraction layer includes a camera algorithm library. Specifically, the camera hardware abstraction layer can provide virtual hardware for the camera device. It can also acquire pose data and transmit it to the camera algorithm library. The camera hardware abstraction layer can also be used to calculate the number N of images to be stitched together, and to obtain information from the camera algorithm library.

[0111] The camera algorithm library may include algorithm modules. For example, an algorithm module may be an image denoising algorithm module, that is, an algorithm module used to implement image denoising processing. In some embodiments, the image denoising algorithm module in the camera algorithm library may include sub-functional modules such as a noise intensity acquisition module and a denoising network. Figure 6 (Not shown in the image).

[0112] The Hardware Abstraction Layer (HAL) interacts with various hardware devices in the Hardware Layer based on the kernel layer (also called the driver layer). For ease of explanation, the kernel layer is not shown in the diagram.

[0113] The camera in the hardware layer is used to acquire images. The digital signal processor and graphics processor provide hardware support for the camera algorithm library.

[0114] For ease of understanding, taking a photography scenario as an example, the image denoising method in this application embodiment is briefly described below in conjunction with the software structure of a mobile phone. For details, please refer to [link / reference]. Figure 7 It includes the following steps:

[0115] 1. The "Camera" application sends a photo-taking request to the application framework layer.

[0116] 2. The application framework layer passes the photo-taking request to the hardware abstraction layer.

[0117] 3. The hardware abstraction layer sends the image capture parameters to the camera.

[0118] 4. The camera returns the image acquisition results.

[0119] 5. The virtual camera device in the hardware abstraction layer indicates that the noise intensity acquisition module in the image denoising algorithm module acquires the noisy image.

[0120] 6. The noise intensity acquisition module extracts prior noise intensity information from the noisy image and then inputs the extracted prior noise intensity information and the noisy image into the denoising network.

[0121] 7. After denoising the noisy image, the denoising network returns the denoised image to the application framework layer.

[0122] 8. The application framework layer further returns the denoised image to the "Gallery" application in the application layer.

[0123] It should be understood that the "Gallery" app can save denoised images.

[0124] Next, the image denoising method in the embodiments of this application will be described in more detail with reference to the accompanying drawings.

[0125] It should be understood that the image denoising method in the embodiments of this application can denoise more accurately and reduce the impact of denoising on image quality. The improvement points include at least the following two stages: Stage 1: Obtain more accurate noise intensity information 2; Stage 2: Input the noise intensity information 2 together with the noisy image / image to be denoised (hereinafter referred to as "Image 1") into the denoising network.

[0126] Phase 1:

[0127] In Phase 1, the acquisition of the aforementioned noise intensity information 2 may include either Method 1 or Method 2:

[0128] Method 1: Obtain noise intensity information directly from auxiliary equipment.

[0129] In this context, "auxiliary device" refers to any device other than an electronic device. The electronic device refers to the device that performs the image denoising method described in this embodiment. Specifically, the auxiliary device can first adjust the noise intensity information 1 corresponding to each pixel in the image 1 to be denoised according to the denoising intensity required to obtain noise intensity information 2. The electronic device can directly obtain this noise intensity information 2 from the auxiliary device.

[0130] The following section will focus on method 2, namely, the steps involved in obtaining noise intensity information 2 by the electronic device itself.

[0131] Method 2: Adjust the noise intensity information 1 to obtain noise intensity information 2.

[0132] The specific processing steps in Method 2 may include:

[0133] (1) Noise intensity estimation.

[0134] The noise intensity information 1 can be obtained in either method a or method b:

[0135] In method a, the electronic device can perform noise intensity estimation processing on image 1 to estimate noise intensity information 1.

[0136] In method b, the electronic device can also directly obtain the estimated noise intensity information from the auxiliary device, without the electronic device itself performing noise intensity estimation processing.

[0137] For example, in method a, the specific processing of noise intensity estimation of image 1 by the electronic device includes: the electronic device using a mathematical model to model the noise in image 1 to obtain a noise model, and estimating the noise intensity of image 1 based on the noise model to obtain noise intensity information 1. For example, the electronic device can construct a Gaussian noise model or a Poisson-Gaussian noise model, etc., for the noise in image 1.

[0138] It should be noted that when there is only one image 1, noise intensity can be estimated for that image 1 to obtain noise intensity information 1. In some cases, there may be multiple images 1, i.e., multiple images are input to the denoising network to increase the amount of information input to the denoising network, enabling the denoising network to output a clearer image. Therefore, when there are multiple images 1, the processing methods can be divided into the following categories:

[0139] Processing method 1: Noise intensity estimation is performed on only one image 1.

[0140] For example, the first frame (i.e., the reference frame) of multiple images 1 can be selected for noise intensity estimation to obtain noise intensity information 1. Subsequently, for each image 1, the denoising strength corresponding to the pixel in each image 1 can be combined with the characteristics of the pixel in each image 1. Then, based on the denoising strength, the following step (2) is performed on the basis of the noise intensity information 1 corresponding to the first frame image 1 to adjust the noise intensity to obtain the noise intensity information 2 corresponding to each image 1.

[0141] Processing method 2: Noise intensity estimation is performed on some special images 1.

[0142] Some special images 1 can be images 1 with relatively representative exposure times. For example, normal exposure frames, short exposure frames, or long exposure frames. Then, for these special images 1, noise intensity estimation is performed to obtain their respective noise intensity information 1. Furthermore, for images 1 that are adjacent to or close to these special images 1, the noise intensity information 1 corresponding to the special images 1 can be used to perform the following step (2) to adjust the noise intensity.

[0143] Processing method 3: Perform noise intensity estimation on all images 1 separately.

[0144] Specifically, noise intensity is estimated for each image 1 to obtain noise intensity information 1 corresponding to each image 1. In this way, each image 1 can perform the following step (2) to adjust the noise intensity based on the denoising strength corresponding to the pixels in the image 1 and using the noise intensity information 1 corresponding to the image 1 itself.

[0145] (2) Noise intensity adjustment.

[0146] For each pixel, the electronic device can adjust the noise intensity information 1 corresponding to that pixel according to the noise reduction level of that pixel, and obtain the noise intensity information 2 corresponding to that pixel.

[0147] In some embodiments, the electronic device can multiply the denoising intensity corresponding to each pixel by the noise intensity information 1 corresponding to that pixel to obtain noise intensity information 2. For example, the denoising intensity can range from (0, N], where N is a preset empirical value that is positive; that is, N cannot be too large, thus giving the final adjusted noise intensity information 2 a certain constraint, preventing it from becoming too large and deviating from reality. For example, N can be 2.

[0148] In some embodiments, the method for obtaining the noise reduction level may include method A or method B:

[0149] Method A: Determine the noise reduction level for each pixel based on its quantization value.

[0150] The quantization value corresponding to a pixel represents the amount of texture information associated with that pixel. For example, the magnitude of the quantization value for each pixel is positively correlated with the amount of texture information associated with that pixel. The texture information associated with a pixel can be the texture information within the pixel's neighborhood.

[0151] In method A, the specific processing for determining the denoising intensity includes: (1) obtaining the quantization value corresponding to each pixel in image 1; (2) determining the denoising intensity corresponding to each pixel based on the quantization value corresponding to each pixel in image 1.

[0152] Specifically, the magnitude of the quantization value corresponding to each pixel is negatively correlated with the magnitude of the denoising intensity corresponding to that pixel. That is, the larger the quantization value, the smaller the denoising intensity corresponding to that pixel, and vice versa.

[0153] In method A above, the electronic device can relatively reduce the noise intensity information 1 in areas with more texture information and relatively increase the noise intensity information 1 in areas with less texture information according to the denoising intensity corresponding to the pixel. Therefore, in subsequent denoising processing, it can avoid or mitigate over-denoising in areas with more texture information and avoid or reduce insufficient denoising intensity in areas with less texture, thus removing pseudo-textures that should not exist in areas with less texture.

[0154] Method B: Determine the noise reduction level for each pixel based on its quantization value and pixel attribute value.

[0155] Pixel attribute values ​​are the specific feature values ​​of pixel attribute parameters. Pixel attribute parameters are attribute parameters used to describe the characteristics of each pixel in an image.

[0156] In some embodiments, the pixel attribute values ​​of a pixel may include at least one of the following: a brightness value or a color attribute value corresponding to the pixel. The color attribute values ​​may include hue, saturation, and brightness.

[0157] It should be understood that in Method B, when determining the denoising intensity corresponding to a pixel, in addition to considering the quantization value used to quantize the amount of texture information, the pixel attribute value of the pixel is also considered. This allows for the consideration of more dimensions of information when determining the denoising intensity, thus enabling not only differentiated denoising between areas with less and more texture, but also differentiated denoising between areas with different pixel attribute values. This makes the denoised image more accurately represent the actual environment or better meet actual needs, thereby improving the accuracy of the denoising intensity.

[0158] Specifically, in method B, the specific processing for determining the denoising intensity corresponding to each pixel includes: for each pixel, the electronic device can calculate a denoising intensity based on the quantization value corresponding to that pixel, and calculate another denoising intensity based on the pixel attribute value of that pixel. Then, the various denoising intensities are combined to obtain the final denoising intensity corresponding to that pixel. Taking the pixel attribute value as a brightness value as an example, in method B, for each pixel in the image 1 to be denoised, the electronic device can calculate denoising intensity 1 (i.e., the first sub-denoising intensity) based on the quantization value corresponding to that pixel, and calculate denoising intensity 2 (i.e., the second sub-denoising intensity) based on the brightness value of that pixel. The final denoising intensity corresponding to that pixel is determined based on denoising intensity 1 and denoising intensity 2.

[0159] In some examples, the brightness value of a pixel can be positively correlated with the denoising intensity 2, or negatively correlated. There is no limitation on this; it can be flexibly set according to actual image processing needs. In other words, the relationship between brightness value and denoising intensity is adjustable, thus meeting any or most image processing requirements. It should be understood that regardless of the image processing need, it is undesirable to confuse bright and dark areas during denoising. Therefore, this solution can differentiate the denoising intensity based on the difference between bright and dark areas, thereby differentiating denoising based on actual needs. This improves the accuracy of denoising processing for any image processing requirement, enhances the image quality after denoising, and ultimately improves the user experience.

[0160] To make it easier to understand, let's take the example of a positive correlation between brightness value and denoising intensity. In Image 1, the darker areas correspond to a lower denoising intensity, while the brighter areas correspond to a higher denoising intensity. This allows the darker areas to retain more texture details and the brighter areas to be smoother after denoising. As a result, the denoised image is closer to the actual environment and improves the user experience.

[0161] As mentioned above, both method A and method B involve the quantization value corresponding to each pixel. The following will explain in detail: (i) the specific form of the quantization value of each pixel, (ii) the calculation method of the quantization value of each pixel, and (iii) how to determine the denoising intensity of each pixel based on its quantization value.

[0162] (a) The specific form of the quantization value for each pixel:

[0163] In some embodiments, the quantization value corresponding to each pixel includes: the pixel value variance or the gradient value corresponding to the pixel.

[0164] (1.1) The following is an explanation of the pixel value variance:

[0165] Here, the variance of the pixel value corresponding to each pixel is the variance of the pixel values ​​of multiple pixels in the pixel's neighborhood. The neighborhood of a pixel is the area within a preset range centered on that pixel. For example, the neighborhood can be a 5x5 or 7x7 area centered on the pixel.

[0166] It should be understood that the pixel value variance is calculated based on the variance of the pixel values ​​of neighboring pixels, thus accurately reflecting the amount of texture information. Therefore, based on the pixel value variance of each pixel, the corresponding noise reduction intensity can be assigned to each pixel.

[0167] For example, the denoising intensity is negatively correlated with the pixel value variance. For instance, pixels with larger pixel value variances correspond to more texture information (i.e., more texture information in the pixel's neighborhood), so a smaller denoising intensity can be applied to avoid over-denoising textured areas. Conversely, pixels with smaller pixel value variances correspond to less texture information (i.e., less texture information in the pixel's neighborhood), so a larger denoising intensity can be applied to avoid insufficient denoising in flat or near-flat areas, resulting in pseudo-textures.

[0168] (1.2) The gradient values ​​are explained as follows:

[0169] The gradient value corresponding to a pixel is used to reflect the rate of change of the pixel value of a pixel in the spatial domain in image 1.

[0170] It should be understood that since gradient values ​​reflect the changes in pixel values ​​in the spatial domain, they can reflect texture information within a certain area. Therefore, based on the gradient values ​​of each pixel, a corresponding denoising intensity can be assigned to each pixel. For example, the magnitude of the denoising intensity is negatively correlated with the magnitude of the gradient value.

[0171] It should be noted that the quantized value corresponding to a pixel is not limited to the pixel value variance or the gradient value corresponding to the pixel. It can also include values ​​calculated by other methods, as long as they can quantify the amount (or quantity) of texture information corresponding to the pixel, so as to distinguish regions with different texture information in Image 1. For example, a pre-trained texture statistics network model can be used to perform texture information statistics on Image 1 to quantify the texture information corresponding to each pixel in the image.

[0172] (ii) How to calculate the quantization value of each pixel:

[0173] (2.1) The calculation method for pixel value variance can include calculation method 1 or calculation method 2:

[0174] Calculation Method 1: Calculate the neighborhood variance pixel by pixel for the entire image.

[0175] It should be understood that the electronic device can directly calculate the neighborhood variance pixel by pixel for image 1. The electronic device can also smooth image 1 and calculate the neighborhood variance pixel by pixel for the smoothed image 3 (i.e., the smoothed image).

[0176] Specifically, the electronic device can treat the image 1 or image 3 to be denoised as a whole and iterate through the pixels in image 1 or image 3 one by one. For each pixel traversed, the variance of the pixel value in the neighborhood of the pixel in image 1 or image 3 is calculated to obtain the variance of the pixel value corresponding to the pixel (i.e., the calculation result of the neighborhood variance).

[0177] The main purpose of smoothing is to reduce noise, distortion, or details in an image, making it visually smoother. For example, an electronic device can use traditional denoising or filtering algorithms to smooth image 1, making it easier to calculate the neighborhood variance of pixels more accurately based on the smoothed image 3.

[0178] In some embodiments, variance calculation using calculation method 1 can also be combined with semantic segmentation. After variance calculation using calculation method 1 and semantic segmentation of image 1 or image 3, the pixel variance value of each pixel in each region after semantic segmentation can be determined. Then, according to the steps described in (3.1) - Processing 1 - Example 2 below, the denoising intensity of each region can be differentiated based on the pixel variance value of each pixel in each region after semantic segmentation. It should be understood that variance calculation using calculation method 1 and semantic segmentation can be processed in parallel or sequentially, without limitation.

[0179] For ease of understanding, combined with Figure 8A A schematic illustration of semantic segmentation is provided. Figure 8A The illustration shows an outdoor image that includes multiple elements such as greenery, people, and sky. Electronic devices can first use perceptual algorithms to perform semantic segmentation on the entire image, dividing it into multiple sub-regions such as greenery region 1, greenery region 2, human figure region, and sky region.

[0180] Calculation Method 2: After semantic segmentation, the neighborhood variance is calculated pixel by pixel within the region.

[0181] Specifically, the electronic device can also semantically segment Image 1 or Image 3 into multiple regions and calculate the neighborhood variance pixel-by-pixel for each region. It should be understood that calculation method 2 no longer calculates the neighborhood variance pixel-by-pixel for the entire image (Image 1 or Image 3), but instead first performs semantic segmentation on the entire image. Then, it treats each region obtained from the semantic segmentation as a whole and calculates the neighborhood variance pixel-by-pixel within each region. This effectively refines the granularity of variance calculation and improves the accuracy of calculating the variance of each pixel value.

[0182] by Figure 8BTaking the green plant area 1 as an example, the calculation method 2 is illustrated. For instance, for pixel A in the green plant area 1, the neighborhood 801 corresponding to pixel A can be determined. Then, the variance of the pixel value of each pixel in the neighborhood 801 is calculated to obtain the variance of the pixel value corresponding to pixel A. A large pixel value variance indicates that the pixel is located in a textured area (or an area with more texture), while a small pixel value variance indicates that the pixel is located in a flat area (or an area with less texture). Thus, by measuring the magnitude of the pixel value variance, areas with more texture (such as the stems of green plants) and areas with less texture (such as empty spaces) in the green plant area 1 can be accurately distinguished. For example, the pixel value variance of pixels at the stems of green plants is greater than the pixel value variance of pixels at the empty spaces.

[0183] It should be understood that after calculating the pixel variance value of each pixel in each region after semantic segmentation using calculation method 2, the denoising intensity in each region can be differentiated according to the pixel variance value of each pixel in each region after semantic segmentation, following the steps described in (3.1)-Process 1-Example 2.

[0184] (2.2) When the quantized value is a gradient value, the gradient value corresponding to the pixel can be calculated by: using the difference method of adjacent pixel values ​​to calculate the gradient value corresponding to each pixel in the image, or using the edge detection operator to perform horizontal and vertical convolution operations on the image, and merging the gradient values ​​in the horizontal and vertical directions to obtain the gradient value corresponding to each pixel.

[0185] (III) How to determine the noise reduction level of a pixel based on the quantization value of each pixel.

[0186] (3.1) The specific processing for determining the denoising intensity of a pixel based on the pixel value variance can include any of the following processing methods:

[0187] Solution 1:

[0188] Electronic devices have a pre-set set of variance thresholds. There can be one or more sets of variance thresholds. Process 1 can be illustrated with the following two examples.

[0189] Example 1: Set a set of variance thresholds.

[0190] It should be understood that a set of variance thresholds may include one or more (at least two) variance thresholds, each with a corresponding denoising intensity. When there is only one variance threshold, different denoising intensities can be assigned to those greater than or equal to that threshold. In this application, the number of variance thresholds or the level of denoising intensity in a set of variance thresholds are not specifically limited. The following will focus on how to determine the denoising intensity when the set of variance thresholds includes multiple variance thresholds.

[0191] For example, each variance threshold set includes multiple variance thresholds, each with its own corresponding denoising intensity. The electronic device can determine the denoising intensity for a pixel based on the comparison results between the pixel value variance and each variance threshold. For instance, the variance threshold set is {V1, V2, V3}, which includes three variance thresholds: V1, V2, and V3, where V1 to V3 progressively increase. V1 corresponds to denoising intensity W1, V2 to W2, and V3 to W3, where W1 to W3 progressively decrease. The electronic device can compare the pixel value variance v of pixel A with V1, V2, and V3 respectively. If the pixel value variance v of pixel A is greater than V3, then pixel A is assigned a denoising intensity W3; if V1 < pixel value variance v ≤ V2, then pixel A is assigned a denoising intensity W2; and if the pixel value variance v ≤ V1, then pixel A is assigned a denoising intensity W1.

[0192] In the above scheme, multiple variance thresholds are set in the variance threshold set, and corresponding denoising intensities are set for each variance threshold. This allows for fine-grained differentiation of the denoising intensity corresponding to each pixel, making the denoising intensity more precise and accurate, thereby improving the accuracy of subsequent denoising processing. Furthermore, in the above processing 1, both the variance threshold and the denoising intensity are configurable and adjustable, thus achieving adjustable denoising in the subsequent process, making the denoised image more in line with the requirements.

[0193] Example 2: Multiple sets of variance thresholds are set.

[0194] For example, different variance threshold sets can be pre-defined for different region types, such as vegetation, sky, and human figures. It should be understood that the variance threshold values ​​in the different variance threshold sets set for different region types can vary.

[0195] Therefore, when the entire image is semantically segmented into multiple regions, for each region, after determining the variance of the pixel values ​​of the pixels within that region, a set of variance thresholds corresponding to the region type to which that region belongs can be obtained. Then, the variance of the pixel values ​​of each pixel within that region can be compared with the variance thresholds in this set of variance thresholds, and the denoising intensity corresponding to that pixel can be determined based on the comparison result. It should be understood that how to compare with the variance thresholds to determine the denoising intensity can be found in the specific description of processing 1 above.

[0196] To facilitate understanding, we will now combine Figure 8C Let's illustrate with examples. From... Figure 8C It can be seen that the image is divided into three region types: green plant region, sky region, and portrait region. Different regions correspond to different variance threshold sets; that is, the variance threshold sets 1, 2, and 3 for the green plant region, sky region, and portrait region are not the same. Since green plant region 1 and green plant region 2 belong to the same region type (both are green plant regions), they correspond to the same variance threshold set 1. Each variance threshold set has its own corresponding denoising strength set. It should be understood that because the texture information and pixel value sizes differ significantly within different types of regions, it is sufficient to differentiate the variance threshold sets corresponding to each region type; there is no restriction on the denoising strength sets for different variance threshold sets. That is, the denoising strength sets for different variance threshold sets can be the same or different. For example, Figure 8C The noise reduction intensity set 1, noise reduction intensity set 2, and noise reduction intensity set 3 can be the same or different.

[0197] To illustrate this more clearly, let's take two region types—green vegetation and sky—as examples. Since green vegetation areas have more texture detail, and it's generally desirable to retain more of this detail, while sky areas have less texture detail, and it's generally desirable to achieve a higher level of noise reduction and smoother texture, different noise reduction levels can be assigned to the same pixel value in both regions based on different configured variance threshold sets. For example, pixels with a value greater than 230 in the green vegetation area might receive a noise reduction level of 0.6, while pixels with a value less than 30 in the sky area could receive a noise reduction level of 0.6 or even higher. This differentiated noise reduction level for different regions is achieved by flexibly configuring the corresponding variance threshold set and noise reduction level for different region types. This flexibility and adjustability make it suitable for any scenario.

[0198] In Example 2, corresponding variance threshold sets are configured based on different region types, and corresponding denoising intensity is specifically configured for each variance threshold in the variance threshold set. This allows for a more accurate determination of the denoising intensity for pixels within each region, thereby improving the accuracy of subsequent denoising.

[0199] Solution 2:

[0200] Electronic devices can also acquire pre-configured mapping relationships, such as pre-fitted mapping formulas. Based on these mapping relationships, the pixel value variance of each pixel can be mapped to the corresponding denoising intensity, achieving fine-grained, pixel-level denoising intensity control and improving the accuracy of subsequent denoising.

[0201] It should be understood that in calculation method 2, each region type has a corresponding variance threshold.

[0202] (3.2) The specific processing for determining the denoising intensity of a pixel based on the gradient value is as follows:

[0203] Similarly, electronic devices can pre-set the correspondence between gradient thresholds and denoising intensity, and determine the denoising intensity corresponding to the gradient value of each pixel based on this correspondence. The specific principle is the same as described in (3.1) above, and will not be repeated here.

[0204] The previous section introduced how methods A and B determine the denoising intensity of each pixel based on its quantization value for stage one. However, in method B, in addition to calculating the denoising intensity based on the pixel's quantization value, it also calculates the denoising intensity based on the pixel's attribute value, and combines the two calculated denoising indices to obtain the final denoising intensity. The following section will use a luminance value as an example to illustrate how the denoising intensity is determined based on the pixel attribute value.

[0205] Specifically, the electronic device can convert Image 1 or Image 3 into a YUV image. YUV is a color encoding method; "Y" represents luminance (or luma), which is the grayscale value, while "U" and "V" represent chrominance (or chroma). The electronic device can extract information from the Y channel of the YUV image to obtain the luminance value of each pixel. For example, after extracting the Y channel information, the electronic device can obtain a luminance map. The size of the luminance map is the same as that of Image 1 or Image 3, meaning that the luminance map corresponds one-to-one with the pixels of Image 1 or Image 3. Therefore, the luminance value corresponding to each pixel in the luminance map is the same as the luminance value corresponding to each pixel in Image 1 or Image 3. Furthermore, the electronic device can determine the denoising level corresponding to each pixel based on the luminance value corresponding to each pixel in the luminance map.

[0206] The electronic device can employ either processing A or processing B to determine the noise reduction level corresponding to each pixel based on its brightness value. The specific processing steps are as follows:

[0207] Solution A:

[0208] Electronic devices have a pre-defined set of brightness thresholds, which may include one or more (at least two) brightness thresholds, each with a corresponding noise reduction level. When there is only one brightness threshold, different noise reduction levels can be assigned to brightness values ​​greater than or equal to that threshold. This application does not specifically limit the number of brightness thresholds or the level of noise reduction level. The following section will focus on how to determine the noise reduction level when the brightness threshold set includes multiple brightness thresholds.

[0209] Specifically, the set of brightness thresholds includes multiple brightness thresholds, each with its own corresponding denoising intensity. The electronic device can determine the denoising intensity for a pixel based on the comparison between its brightness value and each brightness threshold. For example, the brightness thresholds might be Y1, Y2, and Y3, where Y1 increases progressively to Y3. Y1 corresponds to a denoising intensity w1, Y2 to w2, and Y3 to w3, where w1 decreases progressively to w3. The electronic device can compare the brightness value Y of pixel A with Y1, Y2, and Y3 respectively. If the brightness value Y of pixel A is greater than Y3, then a denoising intensity w3 is assigned to pixel A; if Y1 < brightness value Y ≤ Y2, then a denoising intensity w2 is assigned to pixel A; and if brightness value Y ≤ Y1, then a denoising intensity w1 is assigned to pixel A.

[0210] In the above scheme, multiple brightness thresholds are set in the brightness threshold set, and corresponding denoising intensities are set for each brightness threshold. This allows for fine-grained differentiation of the denoising intensity corresponding to each pixel, making the denoising intensity more precise and accurate, thereby improving the accuracy of subsequent denoising processing. Furthermore, in the above processing A, both the brightness thresholds and denoising intensities are configurable and adjustable, thus achieving adjustable denoising and making the denoised image more suitable for the requirements.

[0211] Process B:

[0212] Electronic devices can also obtain pre-configured mapping relationships, such as pre-fitted mapping formulas. Based on these mapping relationships, the brightness value of each pixel can be mapped to the corresponding denoising intensity, achieving fine-grained, pixel-level denoising intensity control and improving the accuracy of subsequent denoising.

[0213] To facilitate understanding, we will now combine Figure 9The overall processing flow of Phase 1 (obtaining more accurate noise intensity information 2 / prior noise intensity information) is explained.

[0214] See Figure 9 In Phase One, the electronic device can perform the following processes:

[0215] 1. Estimate the noise intensity of the noisy image to obtain the noise intensity. Figure 1 .

[0216] 2. Smooth the noisy image to obtain a smooth image.

[0217] 3. Calculate the neighborhood variance pixel by pixel for the smoothed image and generate a variance map.

[0218] 4. Use a perceptual algorithm to perform semantic segmentation on the smooth image to divide the smooth image into multiple regions.

[0219] For example, an image can be divided into multiple regions, such as region 1, region 2, region 3, etc. More specifically, using... Figure 8A Taking an image as an example, an image can be divided into multiple regions such as green plant region 1, green plant region 2, human figure region, and sky region.

[0220] 5. Based on the segmented regions and variance map, calculate the denoising intensity 1 corresponding to each pixel in each region.

[0221] It should be understood that the noise reduction level of 1 is determined based on the variance of the pixel values ​​of each pixel.

[0222] Specifically, for each segmented region, the variance of the pixel value corresponding to each pixel in the variance map within that region is determined, as well as the set of variance thresholds corresponding to the region type. Then, the electronic device can compare the variance of the pixel value corresponding to each pixel in that region with each variance threshold in the set of variance thresholds, and based on the comparison results, determine the denoising intensity 1 corresponding to each pixel in that region.

[0223] It should be noted that each variance threshold set has a corresponding denoising intensity set; that is, each variance threshold in each variance threshold set has a corresponding denoising intensity, thus forming a denoising intensity set. The correspondence between the variance threshold set and the denoising intensity set is configurable and adjustable.

[0224] 6. Generate a brightness map based on the smoothed image.

[0225] 7. Calculate the denoising intensity based on the brightness value of each pixel in the brightness map to determine the denoising intensity 2 corresponding to each pixel.

[0226] 8. Determine the overall noise reduction intensity corresponding to each pixel based on noise reduction intensity 1 and noise reduction intensity 2.

[0227] 9. Using the overall denoising effort as a weight, adjust the estimated noise intensity. Figure 1 The noise intensity information (i.e., noise intensity information 1) in the image is adjusted to obtain the adjusted noise intensity information (i.e., noise intensity information 2) for each pixel.

[0228] 10. Generate prior noise intensity based on the adjusted noise intensity information. Figure 2 .

[0229] Next, we will introduce stage two (that is, inputting noise intensity information 2 along with image 1 into the denoising network):

[0230] Specifically, when inputting noise intensity information 2 and the image to be denoised 1 into the denoising network, the two can be superimposed first to generate a superimposed image, and then the superimposed image can be input into the denoising network.

[0231] It should be understood that, depending on the number of images 1, the superposition of noise intensity information 2 and image 1 can be divided into single image superposition and multi-image superposition.

[0232] (a) Single image overlay:

[0233] It should be understood that single image overlay means that there is only one image 1. After overlaying this image 1 with the noise intensity information 2, image 4 can be obtained.

[0234] For example, noise intensity information 2 can be obtained through noise intensity Figure 2 Characterization. Electronic devices can characterize noise intensity. Figure 2 Image 4 is obtained by overlaying it with image 1 (i.e., the overlaid image). Then, image 4 is input into a denoising network for denoising processing to obtain image 2.

[0235] Image 1 (i.e., the original image to be denoised) can be an RGB image or a RAW image. (The text then abruptly shifts to a discussion of noise intensity.) Figure 2 When overlaying, the various channels in image 1 can be compared with the noise intensity. Figure 2 The channels in the image are superimposed. It should be understood that the noise intensity... Figure 2 The overlay with image 1 is an example of image channel overlay.

[0236] Please see Figure 10 Taking an RGB image as an example, an RGB image includes three channels: color channels R, G, and B, and noise intensity. Figure 2 This includes one channel, which is used to characterize noise intensity information. Electronic devices can correlate the three color channels in RGB with the noise intensity. Figure 2One channel from each image is superimposed to form an image with four channels.

[0237] In some examples, if image 1 is a RAW image, the RAW image can be split into three channels: R, G, and B, and then compared with the noise intensity. Figure 2 One channel from the RAW image is overlaid. In other examples, the RAW image can also be overlaid directly with the noise intensity without channel splitting. Figure 2 One channel in the array is overlaid, and there are no restrictions on this.

[0238] (ii) Overlay of multiple images:

[0239] Multiple image overlay refers to having multiple images 1, each image 1 being overlaid with its corresponding noise intensity information 2 to generate multiple images 5, and then the multiple images 5 being overlaid to generate the final image 4 used as input to the denoising network.

[0240] It should be understood that in order to improve the accuracy of denoising, there can be multiple images 1. That is, to obtain a clear, denoised image, multiple temporally adjacent images 1 can be input into the denoising network to increase the amount of input information, so that the denoising network can refer to more information for denoising.

[0241] In some embodiments, the multiple images 1 can be a preset number of temporally adjacent images, such as three adjacent frames.

[0242] In video recording scenarios, since images are captured in real time, multiple adjacent video frames (i.e., multiple images 1) and their corresponding noise intensities can be compared. Figure 2 The data are combined and used as input (for example, superimposed into a single image and then input into a denoising network), ultimately outputting a clear video frame.

[0243] For a photo-taking scenario, multiple images (1) can also be input. Because in a photo-taking scenario, a single capture command instructs the acquisition of one frame, but the preview image is captured in real-time. Therefore, in addition to the captured image, the real-time preview image can also be obtained, resulting in multiple images (1). Then, these multiple images (1) are compared with their corresponding noise intensities. Figure 2 These images are used as input (for example, superimposed into a single image and then fed into a denoising network) to ultimately produce a clear photograph.

[0244] Specifically, given multiple images 1, the electronic device can process each image 1 according to the method mentioned in Stage 1 above to obtain noise intensity information 2 corresponding to each image 1. Then, each image 1 is superimposed with its corresponding noise intensity information 2 to obtain a sub-overlay image corresponding to each image 1. Further, the electronic device can superimpose multiple sub-overlay images according to the temporal order of their corresponding images 1 to obtain a single superimposed image. The electronic device can then input this superimposed image into a denoising network.

[0245] It should be understood that the specific processing method for superimposing each image with the corresponding noise intensity information 2 can be found in the description of single image superposition above, and will not be repeated here.

[0246] To facilitate understanding, we will now combine Figure 11 The process of overlaying multiple images is illustrated.

[0247] Please see Figure 11 Suppose we have n sequentially adjacent images, F1, F2, ..., Fn. Following the method described in Stage 1, a corresponding noise intensity map, F11, F21, ..., Fn1, is generated for each image. We can first perform a first superposition: F1 and F11 are superimposed to generate F111, F2 and F21 are superimposed to generate F211, and Fn and Fn1 are superimposed to generate Fn11. Then, we perform a second superposition: F111, F211, ..., Fn11 are superimposed to generate a single image, image 4. This image can then be input into a denoising network for noise reduction.

[0248] In some embodiments, after obtaining the denoised image 2 output by the denoising network, further detail enhancement processing can be performed to improve the texture details in image 2 and further mitigate the impact of denoising on image quality. The detail enhancement processing will be described below.

[0249] Specifically, after obtaining image 2 from the denoising network, the electronic device can also acquire detail feature information. This detail feature information is used to characterize the detail features in image 1. The electronic device can then fuse this detail feature information into image 2 to generate a clear image with textured details after denoising, thereby improving image quality.

[0250] In some examples, detailed feature information can be extracted from the noisy image corresponding to image 1.

[0251] Please refer to the details. Figure 12After the denoising network outputs the denoised image (i.e., image 2), the electronic device can subtract the noisy image (i.e., image 1) from the denoising network to obtain a noisy image. This noisy image includes some of the detailed feature information lost in the denoised image. A filtering algorithm can then be used to filter the noisy image to extract detailed feature information and generate a detailed image. For example, the electronic device can filter the noisy image once, discarding low-frequency information and retaining high-frequency information to obtain a detailed image. The filtering algorithm and the size of the filter kernel can be set independently; for example, a Gaussian filtering algorithm with a filter kernel of 3 can be used. Furthermore, the electronic device can superimpose the detailed image and the denoised image to form the final image. It should be understood that the final image contains more valuable texture information than the denoised image, which greatly improves image quality.

[0252] It should be understood that in detail enhancement processing, the degree of detail separation in the noisy image and the degree of superposition of the separated detail features when fused and superimposed onto Image 2 can be controlled by setting parameters. Since the parameters can be adjusted by changing the configuration, detail enhancement processing is an adjustable module. For example, setting the superposition weight and the size of the filter kernel, as described below, demonstrate the adjustable nature of detail enhancement processing.

[0253] As mentioned above, there can be multiple images 1. Therefore, when obtaining a noisy image, we can discuss the following two cases:

[0254] Case 1: The noisy image is a single image.

[0255] In case 1, there is only one noisy image, i.e., only one image 1. In this case, the difference between image 1 and image 2 (i.e., the denoised image) can be directly calculated to obtain the noisy image.

[0256] Case 2: The noisy image consists of multiple images.

[0257] In case 2, there are multiple noisy images 1. Then, the difference between one of the multiple noisy images 1 and image 2 is taken to obtain the denoised image.

[0258] For example, since the multiple images 1 are temporally adjacent images, the electronic device can select a first reference frame from the multiple images 1 and subtract it from image 2. In other examples, the electronic device can also randomly select one image from the multiple images 1 and subtract it from image 2.

[0259] In the following text, combined with Figure 13 The description combines detail enhancement with prior noise reduction processing.

[0260] Please see Figure 13After the denoising network outputs the denoised image (i.e., image 2), the electronic device can subtract the noisy image (i.e., image 1) from the denoising network to obtain the noisy image. Then, the electronic device can input both the noisy image and the denoised image into the detail enhancement module for detail enhancement processing to obtain the final image. It should be understood that the processing performed by the detail enhancement module can include the above-mentioned... Figure 12 The filtering process described herein refers to using filtering algorithms to filter noisy images in order to extract detailed feature information and obtain detailed images.

[0261] It should be understood that when fusing detailed feature information into Image 2, it can be pixel-level fusion. That is, each pixel in Image 2 has corresponding detailed feature information. Then, the detailed feature information corresponding to each pixel can be fused with the pixel value corresponding to that pixel in Image 2, thereby realizing the fusion processing of detailed feature information and Image 2.

[0262] In some examples, the electronic device can directly fuse the detail feature information with image 2. Alternatively, the electronic device can decompose the detail feature information into high-frequency detail feature information and low-frequency detail feature information. When fusing with image 2, the high-frequency detail feature information and low-frequency detail feature information are fused and superimposed with image 2 according to different superposition weights.

[0263] Low-frequency detail features refer to the slowly changing parts of an image with minimal pixel value variation. These typically contain the main structural information of the image and represent a broad range of features. High-frequency detail features, on the other hand, refer to the rapidly changing parts of an image with significant pixel value variation. These are more granular features, reflecting finer structural information within the image.

[0264] The following section will explain how to decompose detailed feature information into high-frequency detailed feature information and low-frequency detailed feature information:

[0265] In some examples, electronic devices can perform high- and low-frequency feature recognition, directly identifying high-frequency and low-frequency detail features from the detail feature information. In other examples, electronic devices can first filter out either high-frequency or low-frequency detail features from the detail feature information according to preset rules, and then subtract one of the filtered high-frequency or low-frequency detail features from the detail feature information to separate the high-frequency and low-frequency detail features.

[0266] In some examples, detailed feature information can be characterized using a detailed image. Electronic devices can break down a detailed image into low-frequency details. Figure 1 and high frequency details Figure 1Specifically, electronic devices can use filtering algorithms to filter detailed images to obtain low-frequency details. Figure 1 Then combine the detailed image and the low-frequency details. Figure 1 By subtraction, high-frequency details are obtained. Figure 1 The filtering algorithm and kernel size can be customized; for example, a Gaussian filter with a kernel size of 3 can be used. In other examples, electronic devices can also use algorithms capable of recognizing high-frequency features to extract high-frequency features from detailed images, thus obtaining high-frequency details. Figure 1 Then combine detailed images and high-frequency details. Figure 1 Subtraction yields low-frequency details. Figure 1 .

[0267] The following section will describe how to fuse high-frequency detail features and low-frequency detail features into image 2:

[0268] The fusion of high-frequency detail feature information, low-frequency detail feature information, and image 2 can also be pixel-level fusion, meaning that each pixel in image 2 has corresponding high-frequency detail feature information or low-frequency detail feature information. Therefore, for each pixel in image 2, the high-frequency detail feature information corresponding to that pixel can be superimposed and fused into image 2 according to superposition weight 1, and the low-frequency detail feature information corresponding to that pixel can be superimposed and fused according to superposition weight 2.

[0269] The following text will focus on low-frequency details. Figure 1 Characterizing low-frequency detail features, with high-frequency details Figure 1 The fusion process will be illustrated using the example of representing high-frequency detailed feature information.

[0270] Electronic devices can capture low-frequency details Figure 1 The pixel value of each pixel in the image is determined by a weighting of 1, and high-frequency details are also considered. Figure 1 The pixel value of each pixel in image 2 is superimposed and merged into image 2 according to the superposition weight 2.

[0271] In some examples, each pixel in image 2 has a corresponding set of superposition weights, namely superposition weight 1 and superposition weight 2. Specifically, the electronic device can, for each pixel A in image 2, adjust the low-frequency details of pixel A. Figure 1 The corresponding pixel value is determined by a superposition weight of 1, and the pixel A is then considered in high-frequency detail. Figure 1 The corresponding pixel value in image A is superimposed on the corresponding pixel value in image B according to a superposition weight of 2. In this way, each pixel has the latest corresponding pixel value, resulting in a clean new image with texture detail. It should be understood that low-frequency details... Figure 1 or high-frequency details Figure 1The pixel value corresponding to a pixel represents the low-frequency or high-frequency detail feature information of that pixel.

[0272] The following section will describe how to determine a set of superimposed weights corresponding to each pixel, which can be determined by at least one of method (1) or method (2):

[0273] Method (1): Determine a set of superposition weights based on the brightness value.

[0274] In method (1), specifically, the magnitude of each set of superposition weights can be determined by the magnitude of the brightness value corresponding to that pixel in image 2. It should be understood that the magnitude of each set of superposition weights can be positively correlated with the magnitude of the brightness value, or negatively correlated. That is, there is no absolute constraint relationship between the magnitude of the brightness value and the magnitude of the superposition weights corresponding to low-frequency / high-frequency details, and the specific settings can be made according to actual needs.

[0275] For example, in image 2, brighter areas (high-brightness areas, such as walls) can have more high-frequency detail features overlaid and fewer low-frequency detail features overlaid to avoid overexposure. Darker areas (low-brightness areas, such as greenery) can have fewer high-frequency detail features overlaid and more low-frequency detail features overlaid. Therefore, in most cases, dark areas do not need to display as much detail to meet the user's visual needs, thus avoiding wasting resources processing dark areas. It should be understood that if the image processing requirement is to sharpen dark areas and blur bright areas, then the opposite can be true: fewer high-frequency detail features and more low-frequency detail features can be overlaid. Conversely, for dark areas, more high-frequency detail features and fewer low-frequency detail features can be overlaid.

[0276] For example, the electronic device can convert image 2 (i.e., the denoised image output by the denoising network) to the YUV domain and extract information from the Y channel to generate a luminance map. The electronic device can assign a set of superposition weights to each pixel based on the luminance value corresponding to each pixel in the luminance map, including superposition weight 1 and superposition weight 2.

[0277] Specifically, the electronic device pre-sets a correspondence between different brightness ranges and different combinations of superposition weights. Each combination of superposition weights is a set of superposition weights, including superposition weight 1 and superposition weight 2. For each pixel, the electronic device can determine the brightness range in which the brightness value corresponding to that pixel in image 2 is located, and then determine the superposition weight combination (i.e., weight1 and weight2) corresponding to that pixel according to the pre-set correspondence.

[0278] Method (2): Determine a set of superimposed weights by combining the semantic segmentation results.

[0279] In method (2), the electronic device can determine the region type of each segmented region based on the semantic segmentation result. When determining a set of superposition weights corresponding to each pixel, the electronic device can determine the size relationship between superposition weight 1 and superposition weight 2 in the set of superposition weights corresponding to the pixel based on the region type of the region where the pixel is located.

[0280] For example, for areas like greenery, which primarily feature large details (such as lines and branches), the superposition weight 1 is less than the superposition weight 2; that is, more low-frequency detail features are superimposed, and fewer high-frequency detail features are superimposed. Conversely, for areas like walls, which primarily feature granular details, the superposition weight 1 is greater than the superposition weight 2; that is, more high-frequency detail features are superimposed, and fewer low-frequency detail features are superimposed.

[0281] It should be noted that electronic devices can also combine methods (1) and (2) to determine a set of superposition weights corresponding to a pixel. That is, when determining a set of superposition weights corresponding to a pixel, both the brightness value of the pixel and the region type of the area to which the pixel belongs can be considered simultaneously. For example, each of these two types of information can determine a set of superposition weights. Then, the two superposition sub-weights 1 corresponding to high-frequency detail feature information in the two sets of superposition weights can be directly arithmetically averaged or weighted averaged to obtain the final superposition weight 1. Similarly, the two superposition sub-weights 2 corresponding to low-frequency detail feature information in the two sets of superposition weights can be directly arithmetically averaged or weighted averaged to obtain the final superposition weight 2.

[0282] The following section will explain how to extract low-frequency details. Figure 1 High-frequency details Figure 1 Overlay into image 2:

[0283] In some embodiments, low-frequency details can be superimposed using either superimposition method 1 or superimposition method 2. Figure 1 High-frequency details Figure 1 Overlay into image 2:

[0284] Overlay method 1:

[0285] Specifically, instead of generating new detail images, one can iterate through each pixel in image 2 step by step, directly extracting low-frequency details. Figure 1 and high frequency details Figure 1 Extract the pixel values ​​corresponding to each pixel, and then extract the low-frequency details. Figure 1 and high frequency details Figure 1 Extracted from Figure 2The pixel value corresponding to each pixel in the image is superimposed and fused with the pixel value corresponding to that pixel in image 2 according to the superposition weight of that pixel to generate a new pixel value for each pixel.

[0286] Overlay method 2:

[0287] Specifically, electronic devices can target low-frequency details. Figure 1 The weighted average of each pixel in the image is 1. This weight is then used to adjust the low-frequency detail map and generate new low-frequency details. Figure 2 For example, electronic devices can capture low-frequency details. Figure 1 The superposition weight 1 for each pixel is related to the low-frequency details of that pixel. Figure 1 Multiplying the pixel values ​​in the image alters low-frequency details. Figure 1 The pixel values ​​of each pixel form new low-frequency details. Figure 2 .

[0288] Similarly, electronic devices can also target high-frequency details. Figure 1 The weighted average of each pixel is 2, which is used to adjust the high-frequency detail map and generate new high-frequency details. Figure 2 For example, electronic devices can capture high-frequency details. Figure 1 The superposition weight 2 corresponding to each pixel is related to the pixel's high-frequency detail. Figure 1 Multiplying the pixel values ​​in the image alters high-frequency details. Figure 1 The pixel values ​​of each pixel form new high-frequency details. Figure 2 .

[0289] Furthermore, electronic devices can capture low-frequency details. Figure 2 and high frequency details Figure 2 This is overlaid onto image 2 to create a clean new image with textured details.

[0290] To facilitate understanding, we will now combine Figure 14 The overall process of detail enhancement processing is explained.

[0291] See Figure 14 In detail enhancement processing, electronic devices can perform the following processes:

[0292] 1. Subtract the noisy image from the denoised image to obtain the noisy image.

[0293] 2. Use filtering algorithms to filter the noisy image to obtain a detailed image.

[0294] 3. Use filtering algorithms to filter the detail image to obtain low-frequency details. Figure 1 .

[0295] 4. Combine high-frequency details with low-frequency details. Figure 1 By subtraction, high-frequency details are obtained. Figure 1 .

[0296] 5. Generate a brightness map based on the denoised image.

[0297] 6. Based on the brightness values ​​of each pixel in the brightness map, determine the superposition weight 1 and superposition weight 2 for each pixel.

[0298] 7. Reduce low-frequency details Figure 1 Adjustments are made according to the superposition weight of 1 for each pixel to obtain low-frequency details. Figure 2 .

[0299] 8. High-frequency details Figure 1 Adjustments are made according to the superposition weight of 2 for each pixel to obtain high-frequency details. Figure 2 .

[0300] 9. Reduce low-frequency details Figure 2 High-frequency details Figure 2 The image is then overlaid onto the denoised image to generate the final image (i.e., the final output image).

[0301] This application also provides a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the functions or steps described in the above method embodiments.

[0302] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform the various functions or steps described in the method embodiments.

[0303] This application also provides a computer program product that, when run on a computer, causes the computer to perform the functions or steps described in the above method embodiments.

[0304] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0305] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0306] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0307] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0308] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0309] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image denoising method, characterized in that, Applied to electronic devices, the method includes: Get the first image; Noise intensity is estimated in the first image to obtain the first noise intensity information corresponding to each pixel in the first image; The first noise intensity information of each pixel is adjusted according to the denoising intensity corresponding to each pixel to obtain the second noise intensity information corresponding to the pixel; wherein, the second noise intensity information corresponding to each pixel in the first image is determined by the denoising intensity corresponding to the pixel and the first noise intensity information of the pixel obtained by noise intensity estimation in the first image; the denoising intensity corresponding to the pixel in the first region of the first image is less than the denoising intensity corresponding to the pixel in the second region of the first image; the first region has more texture information than the second region. The second noise intensity information corresponding to each pixel and the first image are input into a denoising network for denoising processing to obtain the second image.

2. The method according to claim 1, characterized in that, Before adjusting the first noise intensity information of the pixel according to the denoising intensity corresponding to each pixel, the method further includes: The quantization value corresponding to each pixel in the first image is obtained; the quantization value is used to characterize the amount of texture information corresponding to the pixel; the magnitude of the quantization value corresponding to each pixel is positively correlated with the amount of texture information corresponding to the pixel; The denoising intensity corresponding to each pixel is determined based on the quantization value of each pixel in the first image; the magnitude of the quantization value is negatively correlated with the magnitude of the denoising intensity.

3. The method according to claim 2, characterized in that, The quantization value corresponding to each pixel includes the pixel value variance or the gradient value corresponding to the pixel; wherein, the pixel value variance corresponding to each pixel is the variance of the pixel values ​​of multiple pixels in the neighborhood of the pixel.

4. The method according to claim 3, characterized in that, In the case that each pixel corresponds to the quantization value, including the pixel value variance corresponding to the pixel, obtaining the quantization value corresponding to each pixel in the first image includes: Semantic segmentation is performed on the third image to obtain multiple sub-regions; the third image is the first image or an image after smoothing the first image. For each pixel in each sub-region, the denoising intensity corresponding to the pixel is determined based on the variance of the pixel value corresponding to the pixel.

5. The method according to claim 4, characterized in that, The step of determining the denoising intensity for each pixel in each sub-region based on the variance of the pixel value corresponding to the pixel includes: For each sub-region, based on the preset correspondence between the region type and the preset variance threshold, the variance threshold corresponding to the region type of the sub-region is determined; For each pixel in the sub-region, the variance of the pixel value corresponding to the pixel is compared with the determined variance threshold, and the denoising intensity corresponding to the pixel is determined based on the comparison result.

6. The method according to claim 1, characterized in that, Before adjusting the first noise intensity information of the pixel according to the denoising intensity corresponding to each pixel, the method further includes: For each pixel in the first image, a first sub-denoising intensity is determined based on the quantization value corresponding to the pixel; the quantization value is used to characterize the amount of texture information corresponding to the pixel; the magnitude of the quantization value corresponding to each pixel is positively correlated with the amount of texture information corresponding to the pixel. The second sub-denoising intensity corresponding to the pixel is determined based on the brightness value of the pixel. The denoising intensity corresponding to each pixel is determined based on the first sub-denoising intensity and the second sub-denoising intensity corresponding to each pixel.

7. The method according to any one of claims 1-6, characterized in that, The second noise intensity information corresponding to each pixel is represented by a noise intensity map; The step of inputting the second noise intensity information corresponding to each pixel and the first image into a denoising network for denoising processing to obtain the second image includes: The noise intensity map and the first image are superimposed to obtain the fourth image corresponding to the first image; The fourth image is input into the denoising network for denoising processing to obtain the second image.

8. The method according to any one of claims 1-6, characterized in that, The first image consists of multiple images; the multiple first images are temporally adjacent; each first image has a corresponding noise intensity map; The step of inputting the second noise intensity information corresponding to each pixel and the first image into a denoising network for denoising processing to obtain the second image includes: Each first image is superimposed with the corresponding noise intensity map to obtain a fifth image corresponding to each first image; The sixth image is obtained by superimposing the fifth images corresponding to the first images respectively; The sixth image is input into the denoising network for denoising processing to obtain the second image.

9. The method according to any one of claims 1-6, characterized in that, After obtaining the second image, the method further includes: Obtain detailed feature information; the detailed feature information is used to characterize the detailed features in the first image; The detailed feature information is fused into the second image to obtain the seventh image.

10. The method according to claim 9, characterized in that, The detailed feature information is represented by a detailed image; the process of fusing the detailed feature information into the second image to obtain the seventh image includes: The detailed image is split into a first low-frequency detail image and a first high-frequency detail image; each pixel in the second image corresponds to a set of superposition weights, the set of superposition weights includes a first superposition weight and a second superposition weight, and the magnitude of the first superposition weight and the second superposition weight in the set of superposition weights corresponding to each pixel is determined by the magnitude of the brightness value corresponding to the pixel in the second image; For each pixel in the second image, the pixel value corresponding to the pixel in the first low-frequency detail image is superimposed with the pixel value corresponding to the pixel in the second image according to the first superposition weight and the pixel value corresponding to the pixel in the first high-frequency detail image is superimposed with the pixel value corresponding to the pixel in the second image according to the second superposition weight to obtain the seventh image.

11. The method according to claim 10, characterized in that, For each pixel in the second image, the pixel value corresponding to the pixel in the first low-frequency detail image is superimposed with the pixel value corresponding to the pixel in the second image according to a first superposition weight, and the pixel value corresponding to the pixel in the first high-frequency detail image is superimposed with the pixel value corresponding to the pixel in the second image according to a second superposition weight, to obtain the seventh image, including: Multiply the pixel value corresponding to each pixel in the first low-frequency detail map by the first superposition weight corresponding to the pixel to obtain the second low-frequency detail map; The second high-frequency detail map is obtained by multiplying the pixel value corresponding to each pixel in the first high-frequency detail map with the second superposition weight corresponding to the pixel. The second low-frequency detail image and the second high-frequency detail image are superimposed on the second image to obtain the seventh image.

12. The method according to claim 10 or 11, characterized in that, The method further includes: The second image is converted to the YUV domain to extract luminance channel information and obtain a luminance map; For each pixel in the second image, the first superposition weight and the second superposition weight corresponding to the pixel are determined according to the brightness value of the pixel in the brightness map. Wherein, the first superposition weight corresponding to each pixel is used to characterize the superposition degree of the pixel value corresponding to the first low-frequency detail map; the second superposition weight corresponding to each pixel is used to characterize the superposition degree of the pixel value corresponding to the first high-frequency detail map.

13. An electronic device, characterized in that, The electronic device includes at least: a display screen, a memory, and one or more processors; the display screen is used to display images, the memory is used to store computer instructions, and when the one or more processors execute the computer instructions, the electronic device performs the method as described in any one of claims 1-12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-12.

15. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-12.

Images