Image denoising method during shooting and related device
By simultaneously capturing black frames during shooting to obtain online noise parameters, an attention-based image denoising network is constructed, which solves the problem of poor denoising effect caused by the difference between online and offline noise and improves image quality in low-light environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HONOR DEVICE CO LTD
- Filing Date
- 2025-01-18
- Publication Date
- 2026-07-21
AI Technical Summary
In the prior art, when electronic devices capture images in low-light environments, the image denoising effect is poor because there is a difference between online noise and offline calibrated noise.
By simultaneously capturing black frames during actual shooting to obtain online noise parameters, an image denoising network with an attention mechanism is constructed, and feature weights are adjusted to match online noise, thereby improving the denoising effect.
This improves the noise reduction capabilities of image denoising networks in real-world shooting environments, especially enhancing image quality in low-light conditions.
Smart Images
Figure CN122434765A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of terminal devices, and more particularly to an image denoising method and related apparatus during shooting. Background Technology
[0002] Currently, in order to improve the image quality of electronic devices in low-light environments, it is necessary to calibrate the noise of the image sensor offline, synthesize training images based on the offline calibrated noise, and then use the training images to train an image denoising network. Finally, during actual shooting, the captured image is input into the image denoising network to output a denoised image.
[0003] However, due to various factors, there is a difference between the online noise of the image sensor and the offline calibrated noise during actual shooting. Therefore, if an image denoising network trained based on offline calibrated noise is still used for actual captured images, the denoising effect of the output image will be poor.
[0004] Therefore, how to improve image denoising effect is an urgent problem to be solved. Summary of the Invention
[0005] This application provides an image denoising method and related apparatus during image capture. The method is applied to electronic devices including a camera module. The method includes: simultaneously capturing black frames during actual image capture; obtaining online noise parameters based on the black frames; and, based on the error between the online and offline noise parameters, causing the image denoising network to focus on the noise error portion, thereby improving the network's denoising capability.
[0006] In a first aspect, this application provides an image denoising method during shooting, which is applied to an electronic device including a camera. The method includes: shooting a first image and a second image through the camera, wherein the first image is a non-black frame and the second image is a black frame, and the shooting times of the first image and the second image are close; determining a first noise parameter of the camera based on the second image; and denoising the first image using the first noise parameter to obtain a denoised image of the first image.
[0007] By implementing the method provided in the first aspect, online noise during actual shooting is taken into account during the noise reduction process, so that the noise reduction process matches the online environment of actual shooting, the noise reduction effect can be improved.
[0008] In conjunction with the method described in the first aspect, the first image is denoised using the first noise parameter to obtain a denoised image of the first image. Specifically, this includes: denoising the first image using the difference between the first noise parameter and the second noise parameter to obtain a denoised image of the first image, wherein the second noise parameter is the calibration noise parameter of the camera; if the difference is greater, the feature weight corresponding to the region with the greater difference in the first image is greater in the denoising process.
[0009] In this way, during the denoising process, attention is paid to the difference between the online noise parameter (i.e., the first noise parameter) and the offline calibration noise parameter (i.e., the second noise parameter). The greater the difference between the online noise parameter and the offline calibration noise parameter, the greater the optimization space and the better the denoising effect.
[0010] In conjunction with the method described in the first aspect, the first image is denoised using the first noise parameter to obtain a denoised image of the first image. Specifically, this includes: inputting the first image and the first noise parameter into an image denoising network; adjusting the feature weights of the first image in the image denoising network based on the difference between the first noise parameter and the second noise parameter to obtain a denoised image of the first image; wherein the second noise parameter is the calibration noise parameter of the camera.
[0011] In this way, when using an image denoising network, the network takes into account the influence of online noise parameters, thereby improving the denoising effect.
[0012] In conjunction with the method described in the first aspect, the first image is denoised using the first noise parameter to obtain a denoised image of the first image. Specifically, this includes: inputting the first image and the first noise parameter into an image denoising network to obtain a denoised image of the first image; the image denoising network is trained through the following steps: determining an initial image denoising network; determining multiple training ground images and synthesizing each of the training ground images with a second noise parameter to generate a training input image corresponding to each training ground image, wherein the second noise parameter is the calibration noise parameter of the camera; superimposing multiple random values on the second noise parameter to generate multiple third noise parameters; and training the initial image denoising network using the multiple training input images, the multiple training ground images, the second noise parameter, and the multiple third noise parameters to obtain the image denoising network.
[0013] In this way, an image denoising network can be trained based on offline calibrated noise parameters and stored in the electronic device, so that the image denoising network can be used for denoising during subsequent actual shooting.
[0014] In conjunction with the method described in the first aspect, the initial image denoising network is trained using the multiple training input images, the multiple training ground images, the second noise parameter, and the multiple third noise parameters to obtain the image denoising network. Specifically, this includes: during the process of inputting the training input image into the initial image denoising network to obtain the denoised image of the training input image, adjusting the feature weights of the training input image in the initial image denoising network based on the difference between the third noise parameter (a random noise parameter that has an error compared to the offline calibration noise parameter, used to simulate the online noise parameter during actual shooting) and the second noise parameter, until the denoised image output by the adjusted initial image denoising network is close to the training ground image. Then, the adjusted initial image denoising network is determined as the image denoising network.
[0015] In this way, by using offline calibrated noise parameters and random noise parameters that have errors compared to the offline calibrated noise parameters, an attention mechanism is constructed to optimize the image denoising network. This allows the network to learn the denoising ability when there are differences between the noise parameters and the offline calibrated noise parameters. As a result, the network focuses on the difference between online noise and offline calibrated noise. When the online noise in the actual shooting is different from the offline noise, the denoising effect of the network is improved.
[0016] In conjunction with the method described in the first aspect, the image denoising network includes a non-attention module and an attention module. The initial image denoising network is trained using the multiple training input images, the multiple training ground images, the second noise parameter, and the multiple third noise parameters to obtain the image denoising network. Specifically, this includes: sequentially inputting each training input image into the initial image denoising network containing the non-attention module to obtain a denoised image of the training input image; sequentially adjusting the non-attention module in the initial image denoising network until the denoised image output by the adjusted initial image denoising network is close to the training ground image, then determining the adjusted non-attention module as the non-attention module in the image denoising network; sequentially inputting each input image and the third noise parameter into the initial image denoising network containing the attention module and the adjusted non-attention module to obtain a denoised image of the training input image; sequentially adjusting the attention module until the denoised image output by the adjusted initial image denoising network is close to the training ground image, then determining the adjusted attention module as the attention module in the image denoising network.
[0017] In conjunction with the method described in the first aspect, the first noise parameter is a noise parameter that follows a Gaussian distribution, for example, a Gaussian variance. In conjunction with the method described in the first aspect, the non-black frame refers to an image captured by the camera in a non-light-avoided environment, and the black frame refers to an image captured by the camera in a light-avoided environment.
[0018] In this way, noise unrelated to photons entering the camera can be isolated by shooting black frames, and the noise can be characterized by a Gaussian distribution model, thereby obtaining accurate noise parameters and improving the network's denoising capabilities.
[0019] In conjunction with the method described in the first aspect, the second image comprises multiple frames. Specifically, capturing the first and second images using the camera, in conjunction with the method described in the first aspect, includes: capturing the first image using the camera at a first exposure duration, and capturing the second image using the camera at a second exposure duration, wherein the first exposure duration is longer than the second exposure duration, and the second exposure duration is lower than a threshold. This improves the accuracy of acquiring online noise parameters.
[0020] In conjunction with the method described in the first aspect, before capturing the second image through the camera, the method further includes: detecting that the shooting scene is a low-light scene, or detecting that a low-light shooting mode is activated.
[0021] This can improve the quality of images taken in low-light environments.
[0022] In a second aspect, this application provides an electronic device including a camera, one or more memories, one or more processors, and a computer program stored on the memory, the processor executing the computer program to perform the steps described in any of the first aspects.
[0023] Thirdly, this application provides a chip system including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to perform the steps as described in any of the first aspects.
[0024] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, performs the steps as described in any of the first aspects.
[0025] Fifthly, this application provides a computer program product including a computer program that, when executed by a processor, performs the steps as described in any of the first aspects. Attached Figure Description
[0026] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;
[0027] Figure 2 A schematic diagram of the training architecture of an image denoising network provided in an embodiment of this application;
[0028] Figure 3 A schematic diagram of another image denoising network training architecture provided in an embodiment of this application;
[0029] Figure 4 This is a schematic flowchart of an image denoising method during shooting, provided in an embodiment of this application.
[0030] Figure 5 A schematic diagram of the hardware architecture of an electronic device provided in an embodiment of this application;
[0031] Figure 6 This is a schematic diagram of the software architecture of an electronic device provided in an embodiment of this application. Detailed Implementation
[0032] The technical solutions in the embodiments of this application will now be described clearly and in detail with reference to the accompanying drawings. The reference to "embodiment" in this application means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described in this application can be combined with other embodiments.
[0033] The term "user interface (UI)" used in the following embodiments of this application refers to the medium interface through which an application or operating system interacts and exchanges information with a user. It realizes the conversion between the internal form of information and the form that the user can receive. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.
[0034] refer to Figure 1 , Figure 1 An example application scenario diagram is shown.
[0035] like Figure 1As shown, the image denoising method provided in this application is applied in the camera module of an electronic device. The camera module can convert a scene into an image. The camera module includes, but is not limited to, an image sensor, an analog amplifier (AP), an analog-to-digital converter (ADC), and an image signal processor (ISP).
[0036] The process of converting a scene into an image using a camera module includes the following stages.
[0037] (1) Photoelectric Conversion Stage: Light from the scene radiates onto the photosensitive element of the image sensor. The photosensitive element generates electrons through the photoelectric effect and converts them into voltage. Normally, the electrical signal generated by the photosensitive element through the photoelectric effect is relatively weak, especially at pixels where the light radiation is weak. Furthermore, under light-blocking conditions, the photosensitive element is also affected by dark current, generating a weak electrical signal. This dark current is caused by the interference of unstable factors such as temperature and voltage, resulting in a non-zero electrical signal output by the photosensitive element under completely dark conditions.
[0038] It is evident that the noise introduced during the photoelectric conversion stage includes photon noise and dark noise. The photon noise is related to the photons incident on the photosensitive device, while the dark noise is unrelated to the photons input to the photosensitive device and is related to the dark current generated inside the photosensitive device due to temperature influence.
[0039] (2) Electrical signal amplification stage: The analog amplifier amplifies the low voltage generated by the photosensitive device into a high voltage. This is because the sensitivity of the subsequent analog-to-digital converter is limited and it cannot convert the low voltage value in some pixels into the corresponding pixel value. In order to improve the image quality under low light conditions and avoid the loss of dark details in the pixels, it is necessary to use an analog amplifier to amplify the low voltage generated by the photosensitive device into a high voltage.
[0040] It is evident that during the electrical signal amplification stage, the electronic components in the analog amplifier generate noise during operation. This noise is unrelated to the photons incident on the photosensitive device and is related to the performance and operating time of the electronic components in the analog amplifier.
[0041] (3) Analog-to-digital conversion stage: The analog-to-digital converter converts the analog signal (i.e., high voltage) output by the analog amplifier into a digital signal (i.e., pixel value). Digital signals are easy to store and process, but at the same time, when the analog-to-digital converter converts the continuous analog signal into a discrete digital signal, there will be quantization error. Quantization error is caused by the fact that the value of the analog signal cannot be accurately represented by the finite number of bits of the digital signal.
[0042] It is evident that during the analog-to-digital conversion stage, the analog-to-digital converter introduces noise during quantization, and the electronic components inside the converter also introduce noise. This noise is unrelated to the photons incident on the photosensitive device, but is related to the performance and operating time of the electronic components in the analog-to-digital converter.
[0043] (4) Image preprocessing stage: The ISP performs preprocessing such as denoising, format conversion, and geometric correction on the original image (RAW image) represented by pixel values output by the analog-to-digital converter. Optionally, the image denoising method provided in this application is mainly executed in the ISP, for example, by deploying an image denoising network in the ISP and using the image denoising network to obtain a denoised image.
[0044] As can be seen from the foregoing, in electronic device shooting scenarios, the noise introduced can be mainly divided into additive noise and multiplicative noise based on its distribution mathematical model and noise characteristics.
[0045] Additive noise refers to noise independent of the input signal (i.e., photons incident on the photosensitive device). Examples include the dark noise in the photoelectric conversion stage, the noise in the electrical signal amplification stage, and the noise in the analog-to-digital conversion stage, as mentioned earlier. This type of noise always exists regardless of whether light is blocked. Since this type of noise has a higher probability of occurring near the mean and is closer to random phenomena in the real world, a Gaussian distribution mathematical model can be used to characterize the dark noise, the noise in the electrical signal amplification stage, and the noise in the analog-to-digital conversion stage.
[0046] Multiplicative noise refers to noise associated with the input signal (i.e., photons incident on the photosensitive device), such as the photon noise generated during the photoelectric conversion stage as described earlier. Since the incident photons on the photosensitive device are discrete, independent, and random events, a mathematical model of the Poisson distribution can be used to characterize photon noise.
[0047] In one embodiment of this application, a Gaussian-Poisson distribution mathematical model can be used to characterize image noise, specifically as follows: Where x is the noisy image, x * For a noise-free image, k is the photoelectric conversion gain coefficient. Indicates the Poisson distribution. σ represents a Gaussian distribution, where 0 is the mean of the Gaussian distribution, and σ is the mean of the Gaussian distribution. 2 The variance is the Gaussian distribution. Optionally, in some other implementations, only a Gaussian distribution or only a Poisson distribution may be used to characterize the noise in the image.
[0048] In the aforementioned application scenario, an optional image denoising method is as follows: First, the noise parameters of the camera are calibrated offline, and an image denoising network is trained based on the offline-calibrated noise parameters. Subsequently, the trained network is used to denoise the actually captured images. Here, "offline" refers to the stage of rigorously calibrating noise parameters by setting up an experimental environment in non-user-end actual shooting scenarios, while "online" refers to the stage of calibrating noise in user-end actual shooting scenarios without the need for a rigorous experimental environment. Optionally, offline includes the stage of testing electronic devices before they are put on the market and used by users, or offline also includes the stage of returning electronic devices to the factory for testing after they are put on the market and used by users. This application embodiment does not specifically limit the stages referred to as offline and online; the use of offline and online is mainly used to distinguish the environment and methods when calibrating noise. Typically, the noise parameters of the camera can be obtained through various noise calibration methods in the offline stage, and the noise parameters are pre-set in the camera module. This application embodiment does not specifically limit the type of parameters calibrated offline. Optionally, if the offline calibrated noise conforms to a Gaussian distribution, then the noise parameter is a parameter from the Gaussian distribution; if the offline calibrated noise conforms to a Poisson distribution, then the noise parameter is a parameter from the Poisson distribution; if the offline calibrated noise conforms to a Poisson-Gaussian distribution, then the noise parameter may include parameters from both the Poisson and Gaussian distributions. This application will only use noise conforming to a Gaussian distribution as an example for now. For details regarding the image denoising network architecture trained based on the offline calibrated noise parameters, please refer to the following section. Figure 2 The description.
[0049] refer to Figure 2 , Figure 2 An example of a training architecture for an image denoising network is shown.
[0050] like Figure 2 As shown, the training data used in the training architecture of this image denoising network is synthesized based on offline calibrated noise parameters. Specifically, firstly, multiple target images (considered as noise-free images) are selected. Then, the offline noise is synthesized with each of the multiple target images to generate corresponding noisy images. Finally, each noisy image and target image are used as a set of training data to train the image denoising network.
[0051] The training process includes: First, the noisy image is input into the image denoising network as training data, and the network outputs a corresponding denoised image after processing. Second, the image denoising network is progressively optimized to reduce the difference between the denoised image output by the network and the target image corresponding to the input noisy image. Finally, the optimized image denoising network is used as the final trained image denoising network until the difference between the denoised image and the target image meets a preset condition.
[0052] The image denoising network can be a convolutional neural network or other types of networks, and this application embodiment does not limit this.
[0053] based on Figure 2 As shown in the training architecture of an image denoising network, the final trained network only adapts to offline calibrated noise parameters. This is because the training data is synthesized based on these offline calibrated noise parameters. If the online noise parameters differ from the offline calibrated noise parameters during subsequent actual image capture, continuing to use the offline-trained image denoising network will result in poor denoising performance. Furthermore, noise is introduced at each stage of image capture, and some of this noise is strongly correlated with the electronic components used in each stage. The noise introduced by these electronic components changes continuously with usage time and environment. Therefore, in actual online use, the online noise parameters are highly likely to differ from the offline calibrated noise parameters.
[0054] To address the aforementioned technical problems, this application provides an image denoising method and related apparatus during image capture. This method is applied to electronic devices including a camera module. Specifically, the method includes: capturing additional black frames during the actual image capture (non-black frames); obtaining noise parameters for the actual captured image based on the black frames; and using these online noise parameters to denoise the captured image. Specifically, an attention module is constructed based on an image denoising network trained using offline calibrated noise. The actually captured image and the online noise parameters are input into the image denoising network. The attention module can adjust the feature weights of the input image in the image denoising network based on the difference between the online noise parameters and the offline calibrated noise parameters.
[0055] Non-black frames refer to images captured by the camera without avoiding light, while black frames refer to images captured by the camera under conditions of avoiding light.
[0056] The offline calibrated noise parameters include those following a Gaussian distribution, or those following a Poisson distribution. Online noise parameters acquired based on black frames typically only include those following a Gaussian distribution, such as the variance within a Gaussian distribution. This is because when shooting a black frame, the noise does not include noise related to the incident photons, but only noise unrelated to them. Therefore, the online noise parameters acquired based on black frames are Gaussian distributed noise parameters. For details on the types and mathematical distributions of noise in the shooting scene, please refer to the previous section... Figure 1 The description will not be repeated here.
[0057] For details regarding the architecture of the image denoising network trained based on offline calibrated noise, please refer to the previous section. Figure 2 The details of this will not be elaborated here.
[0058] For details on the method of constructing an attention mechanism based on an image denoising network trained on offline calibrated noise, please refer to the following text. Figure 3 The details of this will not be elaborated here.
[0059] For details on the method of using an image denoising network with an attention mechanism, please refer to the following text. Figure 4 The details of this will not be elaborated here.
[0060] It is evident that implementing the image denoising method provided in this application can bring the following beneficial effects:
[0061] (1) An attention mechanism is constructed to optimize the image denoising network, which focuses on the difference between online noise and offline calibration noise. When the online noise and offline noise are different, the denoising effect of the network is improved. In addition, the greater the difference between the online noise parameters and the offline calibration noise parameters, the larger the optimization space and the better the denoising effect.
[0062] (2) By simultaneously capturing black frames when actually capturing images, the hardware environment of the camera module is the same as that of the image capture, thereby ensuring that the online noise obtained based on the black frames is consistent with the online noise when capturing images, and further improving the image denoising effect.
[0063] refer to Figure 3 , Figure 3 An example of another training architecture for an image denoising network is shown.
[0064] like Figure 3 As shown, the training architecture of this image denoising network includes the following stages.
[0065] (1) Construct an image denoising network that includes an attention module.
[0066] Specifically, existing image denoising networks can be reused by adding an attention module, or a new image denoising network with an attention module can be constructed. This application does not impose any limitations on this approach. For information on existing image denoising networks, please refer to the preceding section. Figure 2 As described above, regardless of whether an existing image denoising network is reused or an image denoising network is reconstructed, the embodiments of this application do not show other modules in the image denoising network that are not attention modules, and will not be described in detail here.
[0067] The attention module is used to control the image denoising network to focus on the difference between online noise parameters and offline calibrated noise parameters, thereby improving the network's denoising ability when errors exist between the two. Therefore, the attention module can be determined based on the online noise parameters and the offline calibrated noise parameters. For example, when the offline calibrated noise parameters contain noise parameters that conform to a Gaussian distribution, σattention can be used. 2This represents the offline calibration noise parameter, expressed in σ. ′2 Let represent the random noise parameter that has an error compared to the offline noise parameter. Using this random noise parameter to simulate the online noise parameter in the actual shooting scenario, the attention module can be constructed as: Attention = F·sigmoid(|σ 2 -σ′ 2 |), where F is the feature map of the input image in the network, and sigmoid is a commonly used activation function in networks. When σ 2 With σ′ 2 The larger the difference, the more sigmoid(|σ) becomes. 2 -σ′ 2 The larger the value of |, the greater the weight of F in the network, thus making the network pay more attention to σ. 2 With σ′ 2 The difference, that is, when the online noise parameter σ′ corresponds to a certain region of pixels in the input image. 2 With the corresponding offline calibrated noise σ 2 The greater the difference, the greater the weight of the pixels in that region, and the stronger the noise reduction capability of the optimized network.
[0068] (2) Train an image denoising network that includes an attention module.
[0069] Phase 1: Determine the training data.
[0070] Specifically, the training data includes: multiple random noises, multiple sets of training input images, and training ground image pairs.
[0071] Random noise is used to simulate online noise that differs randomly from the offline calibration noise in actual image capture. Therefore, this application obtains multiple random noises by superimposing multiple random perturbations on the offline calibration noise, which can be mathematically expressed as: σ i ′2 =σ 2 +δ i , where δ i For the i-th random perturbation, σ i ′2 Let be the parameters of the i-th random noise.
[0072] The training input images and ground truth images are the experimental data used to train the image denoising network. Specifically, firstly, multiple target images (considered as noise-free images) are selected as ground truth images. Then, offline calibrated noise is used to synthesize a corresponding noisy image with each target image. Finally, the synthesized noisy image is used as the training input image.
[0073] Phase 2: Training the image denoising network.
[0074] Specifically, firstly, the previously constructed image denoising network including an attention module is used, and the aforementioned noisy image is input into the image denoising network as the training input image, and the aforementioned random noise parameter σ is... i ′2 The input is fed into the image denoising network. Then, the attention module in the image denoising network is based on the parameter σ of the random noise. i ′2 With offline calibration noise parameter σ 2 The feature weights of the input image are adjusted based on the differences in the input image, specifically the weights of the feature map F corresponding to the input image. After a series of processing steps by the image denoising network, a denoised image is output. Finally, the training ground image is used as the ideal denoised image for comparison with the actual denoised image. The image denoising network is then optimized in reverse, including optimizing the attention module, to reduce the gap between the ideal and actual denoised images. This ensures that the denoised image actually output by the optimized network is close to the ideal denoised image. Once the difference between the two meets a preset condition, the optimized network is used as the final learned image denoising network.
[0075] (3) Online application of image denoising network with attention module.
[0076] Phase 1: Online synchronous shooting of non-black frames and black frames.
[0077] Specifically, when a user captures an image (non-black frame) using an electronic device, a black frame is also captured additionally. Synchronous shooting means that the shooting times are close together, with an extremely small and negligible time difference. This ensures that the shooting environment for both the non-black and black frames is consistent, meaning that the Gaussian noise independent of the input photons in both frames is almost identical. A non-black frame refers to an image captured by the electronic device under non-light-protected conditions, such as by opening the shutter (mechanical shutter, and / or electronic shutter) and capturing a color image at a certain exposure time. A black frame refers to a completely black image captured by the electronic device by opening the shutter (mechanical shutter, and / or electronic shutter) and capturing a completely black image at an extremely short exposure time.
[0078] Phase 2: Determine the online noise parameters corresponding to non-black frames based on black frames.
[0079] Specifically, based on the preceding introduction to the types and mathematical distributions of noise, the noise introduced by capturing black frames can be described as Gaussian noise independent of the signal. Therefore, the noise of black frames can be modeled as a Gaussian distribution model using statistical variance. Where, x b The black frame contains Gaussian noise. This represents a Gaussian distribution, where 0 is the mean of the Gaussian distribution. Let Variance be the variance of the Gaussian noise in the black frame. Since the shooting environment of this black frame is the same as that of the non-black frames, the noise parameters corresponding to this black frame, such as the Gaussian variance, are... It can be regarded as an online noise parameter of non-black frames.
[0080] Stage 3: Input the captured non-black frames and the calculated online noise into an image denoising network containing an attention module to output a denoised image.
[0081] Specifically, noise parameters calculated based on black frames. This can be considered as an online noise parameter when shooting non-black frames; therefore, non-black frames can be compared with... The input is fed into an image denoising network, where the attention module can be based on online noise parameters. With offline calibration noise parameter σ 2 The weights of the corresponding feature maps of the input non-black frames are adjusted based on the differences, i.e.: When σ 2 and The greater the difference, the more the attention module increases the weight of the non-black frame feature map F, allowing the network to focus more on these differing parts and thus better capture key information. In other words, the difference between online noise and offline calibration noise reflects changes in noise distribution. Increasing the weight of pixels in the feature map where noise changes allows the network to pay more attention to important features resulting from noise variations, helping the network adapt to new environments and data characteristics more quickly, thereby improving the network's denoising performance.
[0082] refer to Figure 4 , Figure 4 An example of an image denoising method during shooting is shown.
[0083] like Figure 4 As shown, the training architecture of this image denoising network includes the following stages.
[0084] Offline phase (S401-S403): Pre-acquire image denoising network.
[0085] S401 performs offline noise calibration for cameras in electronic devices.
[0086] Specifically, designers can perform offline noise calibration of the camera according to the mathematical model of Gaussian-Poisson distribution, or they can perform offline noise calibration of the camera according to the mathematical model of Gaussian distribution. This application embodiment does not limit this, and you can refer to the existing offline noise calibration methods, which will not be elaborated here.
[0087] In this embodiment of the application, the offline calibrated parameters (e.g., σ) 2 It can also be called the second noise parameter.
[0088] For information on the causes and mathematical distribution types of noise in shooting scenes, please refer to the previous text. Figure 1 The details of this will not be elaborated here.
[0089] S402, synthesizes training data based on offline calibrated noise parameters, and generates random noise parameters that simulate online noise parameters based on offline calibrated noise parameters.
[0090] Specifically, a target image (considered a noise-free image) can be selected first. Then, the target image is synthesized with offline-calibrated noise parameters to obtain a noisy image. Finally, the target image and the corresponding noisy image are treated as a pair of training data. Furthermore, random perturbations are superimposed on the offline-calibrated noise parameters to generate random noise parameters. Since the online noise parameters have a certain error compared to the offline noise parameters, these random noise parameters can be used to simulate subsequent online noise parameters for training the image denoising network, ensuring that a better denoising effect can be output when using online noise parameters later.
[0091] For a more detailed description of synthesizing training data and generating random noise parameters, please refer to the previous section. Figure 3 The details will not be elaborated here.
[0092] In this embodiment, random noise parameters (e.g., σ) that simulate online noise parameters are generated based on offline calibrated noise parameters. ′2 It can also be called the third noise parameter.
[0093] S403, construct an initial image denoising network including an attention module, and train the initial image denoising network using training data and random noise parameters to obtain the image denoising network.
[0094] Specifically, for the architecture of the initial image denoising network and the architecture for training this initial image denoising network, please refer to [reference needed]. Figure 3 And the previous text on Figure 3 The description will not be repeated here.
[0095] S404, The electronic device detects a shooting operation, and in response to the shooting operation, captures an image and simultaneously captures a black frame.
[0096] Specifically, when the electronic device is a mobile phone, the user can use the camera application to take pictures (non-black frames). This shooting operation is, for example, clicking the shutter button in the camera application. In response to this operation, the mobile phone can control the camera to take a picture of the face and control the camera to take a black frame.
[0097] Optionally, the shooting operation can also be other types of operation. When the product type of electronic device is different, the shooting operation may be different. This application embodiment does not limit this.
[0098] Optionally, the number of non-black frames captured can be one or more. If multiple non-black frames are captured, they can be merged into a single HDR image. The number of black frames captured can be one or more.
[0099] Optionally, after the electronic device detects the shooting operation, it can also detect whether the current shooting scene is a low-light scene. If so, it will also capture a black frame at the same time when shooting the image, which will be used for subsequent image denoising. Alternatively, if the electronic device has night mode (low-light mode) enabled in advance, it will directly capture a black frame at the same time after detecting the shooting operation.
[0100] S405 acquires online noise parameters based on captured black frames.
[0101] Specifically, the main noise contained in black frames consists of additive Gaussian noise, including dark current noise, electrical signal amplification stage noise, and readout noise from analog amplifiers. Therefore, the parameters of online noise can be calculated using methods such as direct calculation or the difference method. The direct calculation method involves: first, statistically analyzing each pixel in multiple captured black frame images and calculating its mean and variance. Since a black frame image should theoretically be completely black, the mean reflects any fixed-pattern noise or bias that may exist in the image, while the variance represents the intensity of the Gaussian noise. Generally, the larger the variance, the higher the power spectral density of the Gaussian noise, i.e., the more severe the noise. Then, the Gaussian noise parameters are determined: based on the calculated mean and variance, the mean and variance of the Gaussian noise can be determined. The difference method involves: first, continuously capturing a series of black frame images at the same exposure time to form a black frame sequence; calculating the difference between adjacent frames, i.e., subtracting the previous frame from the next frame, to obtain a difference image sequence. This can eliminate fixed-pattern noise and bias in the image and highlight the variations in Gaussian noise. Then, statistical analysis is performed on the differenced image sequence to calculate its mean, variance and other statistics. Since the differenced image mainly contains the random variation of Gaussian noise, the parameters of Gaussian noise can be estimated through these statistics.
[0102] In this embodiment of the application, online noise parameters (e.g., obtained from black frames during actual shooting) are used. It can also be called the first noise parameter.
[0103] S406 inputs the captured image and acquired online noise parameters into the image denoising network to obtain the corresponding denoised image.
[0104] Optionally, when the captured images contain multiple images, each image can be denoised separately before being fused to generate an HDR image, or multiple images can be fused to generate an HDR image before denoising. This application does not limit this.
[0105] The following section describes the hardware and software architecture of the electronic device involved in this application.
[0106] Electronic devices can be equipped with Or other portable terminal devices with different operating systems, such as cameras, mobile phones, tablets, desktop computers, laptops, handheld computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices and / or smart city devices, etc.
[0107] refer to Figure 5 , Figure 5 An example of an electronic device hardware architecture is shown.
[0108] like Figure 5 As shown, the electronic device may include: a processor 110, an external memory interface 120, an internal memory 121, a camera 130, a display screen 140, and a sensor module 160. The sensor module 160 may include a pressure sensor 160A, a touch sensor 160B, etc.
[0109] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, the electronic device may also include buttons, motors, indicators, and a subscriber identification module (SIM) card interface, etc. As another example, the sensor module may also include: a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a fingerprint sensor, a temperature sensor, a bone conduction sensor, etc.
[0110] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors. The controller may be the nerve center and command center of the electronic device. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.
[0111] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0112] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include inter-integrated circuit (I2C) interfaces, inter-integrated circuit sound (I2S) interfaces, pulse code modulation (PCM) interfaces, universal asynchronous receiver / transmitter (UART) interfaces, mobile industry processor interfaces (MIPI), general-purpose input / output (GPIO) interfaces, subscriber identity module (SIM) interfaces, and / or universal serial bus (USB) interfaces, etc. It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.
[0113] In this embodiment, the processor 110 can receive a shooting operation, control the camera 130 to capture images (non-black frames) and simultaneously capture black frames, and perform noise reduction on the non-black frames based on the captured non-black frames and black frames. For specific noise reduction methods, please refer to the previous section. Figures 3-4 The description of the network used for image denoising will not be repeated here. Optionally, the network used for image denoising can run in the processor 110 or be pre-installed in the ISP, and this embodiment does not limit this.
[0114] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).
[0115] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and double data rate synchronous dynamic random access memory (DDR SDRAM, such as fifth-generation DDR SDRAM, which is generally called DDR5 SDRAM). Non-volatile memory can include disk storage devices and flash memory.
[0116] Random access memory (RAM) can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data. Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into RAM for direct read and write by the processor 110.
[0117] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.
[0118] In this embodiment of the application, the memory described above can be used to store an image denoising network and to store denoised images, etc.
[0119] Electronic devices implement display functions through a GPU, a display screen 140, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 140 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.
[0120] Display screen 140 is used to display images, videos, etc. Display screen 140 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be manufactured using organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes (AMOLEDs), flexible light-emitting diodes (FLEDs), miniled, microLEDs, micro-OLEDs, quantum dot light-emitting diodes (QLEDs), etc. In some embodiments, the electronic device may include one or N displays 140, where N is a positive integer greater than 1.
[0121] In this embodiment, the display screen 140 can be used to display the interface provided by the camera application and the interface provided by the gallery, including but not limited to the preview interface, shooting interface, settings interface, etc.
[0122] Electronic devices can achieve shooting functions through ISP, camera 130, video codec, GPU, display 140 and application processor.
[0123] The ISP is used to process data fed back from the camera 130. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also optimize image noise, such as the image denoising described above. The ISP can also perform algorithmic optimization of image brightness, such as fusing multiple frames to generate a single HDR image. In some embodiments, the ISP can be located within the camera 130 and is collectively referred to as the camera module.
[0124] Camera 130 is used to capture still images or videos. Camera 130 includes the image sensor described above. An object is projected onto a photosensitive element through a lens to generate an optical image. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 130, where N is a positive integer greater than 1.
[0125] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device is selecting a frequency, a DSP can perform a Fourier transform on the frequency energy.
[0126] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0127] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.
[0128] Pressure sensor 160A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 160A can be disposed on display screen 140. There are many types of pressure sensors 160A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 160A, the capacitance between the electrodes changes. The electronic device determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 140, the electronic device detects the intensity of the touch operation based on pressure sensor 160A. The electronic device can also calculate the touch position based on the detection signal from pressure sensor 160A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.
[0129] Touch sensor 160B, also known as a "touch panel," can be located on display screen 140. The touch sensor 160B and display screen 140 together form a touchscreen, also known as a "touchscreen." Touch sensor 160B is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 140. In other embodiments, touch sensor 160B may also be located on the surface of the electronic device, in a different position than display screen 140.
[0130] In this embodiment of the application, the electronic device can detect user operations, such as the operation of taking an image, through the pressure sensor 160A and / or the touch sensor 160B, and transmit them to the processor 110, so that the processor 110 responds to the operation by performing subsequent image taking and image denoising, etc.
[0131] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture Android system as an example to illustrate the software structure of an electronic device.
[0132] Figure 6 This is a schematic diagram of the software architecture of an electronic device according to an embodiment of this application.
[0133] like Figure 6As shown, a layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, [the following is omitted as the text is incomplete and likely refers to a specific implementation or feature]. The system is divided into five layers, from top to bottom: application layer, application framework layer, hardware abstraction layer, kernel layer, and hardware layer.
[0134] The application layer can include a series of application packages. For example, an application package can include applications such as a camera and a gallery.
[0135] The camera application includes, but is not limited to, a photo-taking module and a video-recording module. The photo-taking and video-recording modules can provide various shooting modes. The photo-taking module performs the camera application's photo-taking functions, including outputting a preview interface in photo-taking mode and taking photos after the shooting control is pressed. The photo-taking / video-recording modules include various shooting modes, such as low-light mode (also known as night scene mode), HDR mode, and portrait mode. When the camera application enables low-light shooting mode, the image noise reduction methods described earlier can be applied.
[0136] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications within the application layer. The application framework layer includes predefined functions. For example, it may include camera services, camera devices, and view systems, etc.
[0137] Camera services, also known as camera access interfaces, are used to provide application programming interfaces and programming frameworks for camera applications.
[0138] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.
[0139] The Hardware Abstraction Layer (HAL) is an interface layer located between the application framework layer and the kernel layer, providing a virtual hardware platform for the operating system.
[0140] In this embodiment of the application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.
[0141] The Camera Hardware Abstraction Layer (Camera HAL) provides virtual hardware for both Camera Device 1 and Camera Device 2. It can also acquire pose data and transmit it to the camera algorithm library, as well as retrieve information from the camera algorithm library. CameraHAL is primarily used to control the corresponding sensors to acquire images.
[0142] The camera algorithm library may include one or more algorithm modules, such as motion recognition and AI stereo model algorithm modules, used to implement portrait mode, etc.
[0143] The kernel layer is the layer between hardware and software. It includes drivers for various hardware components. In some embodiments, the kernel layer may include camera device drivers, digital signal processor (DSP) drivers, and image processor (IP) drivers. The camera device driver drives the camera's sensor to acquire images and drives the image signal processor to preprocess the images. The DSP driver drives the DSP to process images. The IIP driver drives the graphics processor to process images. The kernel layer is the layer between hardware and software.
[0144] The hardware layer includes sensors, such as camera 1, camera 2, TOF, multispectral sensors, digital signal processors, and image signal processors.
[0145] The following example, using a scene of capturing a photograph, illustrates the workflow of the electronic device's software and hardware.
[0146] When touch sensor 160B receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a single touch operation as an example, where the corresponding control is the camera application icon, the camera application calls the interface of the application framework layer to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through camera 130.
[0147] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.
[0148] This application also provides an electronic device that may include a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method performed by the electronic device as described in any of the above embodiments.
[0149] This application also provides a chip system including a processing circuit and an interface circuit. The interface circuit is used to receive computer instructions and transmit them to the processing circuit. The processing circuit is used to execute the computer instructions to implement the method performed by the electronic device as in any of the above embodiments.
[0150] This application also provides a chip system including at least one processor for implementing the methods executed by the electronic device in any of the above embodiments. In one possible design, the chip system further includes a memory for storing program instructions and data, the memory being located within or outside the processor.
[0151] A chip system can consist of chips or include chips and other discrete components.
[0152] Optionally, there may be one or more processors in the chip system. The processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.
[0153] Optionally, the chip system may contain one or more memories. These memories may be integrated with the processor or disposed separately; this application does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed on different chips. This application does not specifically limit the type of memory or the arrangement of the memory and processor.
[0154] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.
[0155] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method executed by the electronic device in any of the above embodiments.
[0156] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method executed by the electronic device as described in any of the above embodiments.
[0157] The various embodiments of this application can be combined arbitrarily to achieve different technical effects.
[0158] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).
[0159] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.
[0160] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.
[0161] The terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.
[0162] In summary, the above description is merely an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the disclosure of this application should be included within the scope of protection of this application.
Claims
1. A method for denoising images during shooting, characterized in that, The method is applied to an electronic device including a camera, and the method includes: The camera captures a first image and a second image, where the first image is a non-black frame and the second image is a black frame, and the first and second images are captured at similar times. Determine the first noise parameter of the camera based on the second image; The first image is denoised using the first noise parameter to obtain a denoised image of the first image.
2. The method according to claim 1, characterized in that, The first image is denoised using the first noise parameter to obtain a denoised image of the first image, specifically including: The first image is denoised using the difference between the first noise parameter and the second noise parameter to obtain a denoised image of the first image. The second noise parameter is the calibration noise parameter of the camera. The greater the difference, the greater the feature weight corresponding to the region with the greater difference in the first image during denoising.
3. The method according to claim 1 or 2, characterized in that, The first image is denoised using the first noise parameter to obtain a denoised image of the first image, specifically including: The first image and the first noise parameter are input into an image denoising network. The image denoising network adjusts the feature weights of the first image in the image denoising network based on the difference between the first noise parameter and the second noise parameter to obtain a denoised image of the first image. The second noise parameter is the calibration noise parameter of the camera.
4. The method according to any one of claims 1-3, characterized in that, The first image is denoised using the first noise parameter to obtain a denoised image of the first image, specifically including: The first image and the first noise parameters are input into an image denoising network to obtain a denoised image of the first image; The image denoising network is trained through the following steps: Determine the initial image denoising network; Multiple training ground truth images are determined, and each training ground truth image is combined with a second noise parameter to generate a training input image corresponding to each training ground truth image. The second noise parameter is the calibration noise parameter of the camera. Multiple random values are superimposed on the second noise parameter to generate multiple third noise parameters; The initial image denoising network is trained using the multiple training input images, the multiple training ground images, the second noise parameter, and the multiple third noise parameters to obtain the image denoising network.
5. The method according to claim 4, characterized in that, The initial image denoising network is trained using the plurality of training input images, the plurality of training ground images, the second noise parameter, and the plurality of third noise parameters to obtain the image denoising network, specifically including: In the process of inputting the training input image into the initial image denoising network to obtain the denoised image of the training input image, the feature weights of the training input image in the initial image denoising network are adjusted based on the difference between the third noise parameter and the second noise parameter until the denoised image output by the adjusted initial image denoising network is similar to the training ground image. Then, the adjusted initial image denoising network is determined as the image denoising network.
6. The method according to claim 4 or 5, characterized in that, The image denoising network includes a non-attention module and an attention module. The initial image denoising network is trained using the multiple training input images, the multiple training ground images, the second noise parameter, and the multiple third noise parameters to obtain the image denoising network, specifically including: Each training input image is sequentially input into an initial image denoising network containing a non-attention module to obtain a denoised image of the training input image. The non-attention modules in the initial image denoising network are then sequentially adjusted until the denoised image output by the adjusted initial image denoising network is close to the training ground image. The adjusted non-attention modules are then identified as non-attention modules in the image denoising network. Each input image and the third noise parameter are sequentially input into an initial image denoising network containing an attention module and an adjusted non-attention module to obtain a denoised image of the training input image. The attention module is then sequentially adjusted until the denoised image output by the adjusted initial image denoising network is close to the training ground truth image. The adjusted attention module is then determined as the attention module in the image denoising network.
7. The method according to any one of claims 1-6, characterized in that, The first noise parameter is a noise parameter that follows a Gaussian distribution.
8. The method according to any one of claims 1-7, characterized in that, The non-black frame refers to an image captured by the camera in a non-light-avoiding condition, while the black frame refers to an image captured by the camera in a light-avoiding condition.
9. The method according to any one of claims 1-8, characterized in that, The second image comprises multiple frames.
10. The method according to any one of claims 1-9, characterized in that, Capturing the first image and the second image using the camera specifically includes: capturing the first image using the camera at a first exposure duration, and capturing the second image using the camera at a second exposure duration, wherein the first exposure duration is longer than the second exposure duration, and the second exposure duration is shorter than a threshold.
11. The method according to any one of claims 1-10, characterized in that, Before capturing the second image via the camera, the method further includes: The shooting scene is detected to be in low light, or the low light shooting mode is detected to be enabled.
12. An electronic device, characterized in that, The method includes a camera, one or more memories, one or more processors, and a computer program stored on the memories, the processor executing the computer program to implement the steps of the method according to any one of claims 1-11.
13. A chip system, characterized in that, The chip system includes a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method as described in any one of claims 1-11.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the steps of the method according to any one of claims 1-11.
15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it performs the steps of the method according to any one of claims 1-11.