HDR module single-frame noise reduction method, device and equipment based on deep learning

Abstract

The invention discloses an HDR module single-frame noise reduction method, device and equipment based on deep learning. The method comprises the following steps: acquiring a single-frame image acquired by an HDR module; and inputting the single-frame image into a trained noise reduction neural network model, wherein the trained noise reduction neural network model carries out noise reduction processing on the single-frame image and outputs the single-frame image after noise reduction, wherein the trained noise reduction neural network model is obtained by training based on a data pair composed of a noisy image and a corresponding noiseless image, and the noisy image is obtained by adding noise to the noiseless image based on the brightness of each pixel based on a calibrated noise model of the HDR module. According to the method, the similarity between the generated simulation image and the image actually shot by the HDR module can be ensured, so that the noise reduction performance of the trained noise reduction neural network model can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a single-frame noise reduction method, device and equipment for an HDR module based on deep learning. Background Art

[0002] High Dynamic Range (HDR) images usually have a wide range of brightness and can capture a lot of information from shadow details to highlight details. However, due to the influence of image sensor noise and other environmental factors, HDR images are usually interfered by noise, resulting in reduced image quality and details. Since there are obvious brightness differences in HDR images, noise may have different effects on different brightness areas. Therefore, denoising of HDR images needs to take into account the brightness information in the image and perform corresponding processing according to the brightness levels of different areas.

[0003] However, existing single-frame image denoising methods have difficulty in dealing with nonlinear, highly complex noise and detail loss caused by denoising. Summary of the invention

[0004] The deep learning-based single-frame noise reduction method, device and equipment for the HDR module provided in the embodiments of the present application can solve or partially solve the above-mentioned deficiencies in the prior art or other deficiencies in the prior art.

[0005] According to the first aspect of the present application, a single-frame denoising method for an HDR module based on deep learning is provided, comprising: obtaining a single-frame image captured by an HDR module; and inputting the single-frame image into a trained denoising neural network model, wherein the trained denoising neural network model performs denoising on the single-frame image and outputs the denoised single-frame image; wherein the trained denoising neural network model is trained based on a data pair consisting of a noisy image and a corresponding noise-free image, and the noisy image is obtained by adding noise to the noise-free image based on the brightness of each pixel based on a calibrated noise model of the HDR module.

[0006] In one embodiment of the present application, it also includes: based on multiple frames of original images of the calibration plate under different lighting environments and multiple frames of pure black images under a pure black environment collected by the HDR module, the noise model of the HDR module is calibrated to obtain the correspondence between different pixel brightness and noise intensity; according to the correspondence, noise is added to the noise-free image according to the brightness of each pixel in the noise-free image to obtain the noisy image, and the corresponding noisy image and the noise-free image are combined into a data pair; and the denoising neural network model is trained based on the data pair to obtain the trained denoising neural network model.

[0007] In one embodiment of the present application, based on multiple frames of original images of a calibration plate under different lighting environments and multiple frames of pure black images under a pure black environment collected by the HDR module, the noise model of the HDR module is calibrated to obtain a correspondence between different pixel brightness and noise intensity, including: calibrating the additive noise in the noise model based on the multiple frames of pure black images; and calibrating the multiplicative noise in the noise model based on the multiple frames of pure black images and the multiple frames of original images.

[0008] In one embodiment of the present application, the additive noise includes: at least one of read noise, dark current noise and row noise; and / or the multiplicative noise includes photon scattering noise.

[0009] In one embodiment of the present application, the multiplicative noise in the noise model is calibrated based on the multiple frames of pure black images and the multiple frames of original images, including: determining the mean of each pixel for the multiple frames of pure black images, and determining the position of abnormal pixels based on a preset brightness threshold; subtracting the additive noise from the multiple frames of original images to obtain a third target image, and determining the mean and variance of each pixel for multiple frames of the third target images under each lighting environment; determining the area to be detected in multiple frames of the third target images based on the position of the calibration plate in the image, and obtaining the detection mask of multiple frames of the third target images based on the position of the abnormal pixels; and determining the target pixels in multiple frames of the third target images under different lighting environments based on the detection mask, counting the mean and variance of each pixel in the target pixels, and obtaining the corresponding relationship between different pixel brightness and noise intensity to calibrate the photon scattering noise.

[0010] In one embodiment of the present application, the correspondence between the brightness of different pixels and the noise intensity presents a multi-segment linear relationship.

[0011] In one embodiment of the present application, the multiplicative noise in the noise model is calibrated based on the multiple frames of pure black images and the multiple frames of original images, and also includes: smoothing the multiple linear relationships based on local filtering and local weighted linear regression; and determining the inflection points in the multiple linear relationships after smoothing based on a preset regional threshold, and performing linear fitting based on the inflection points to obtain the noise model corresponding to each linear relationship.

[0012] In one embodiment of the present application, the additive noise in the noise model is calibrated based on the multiple frames of pure black images, including: determining the mean value of each pixel in the multiple frames of pure black images to calibrate the dark current noise, and averaging the means of all pixels to obtain a uniform black level; subtracting the dark current noise from the multiple frames of pure black images and adding the uniform black level to obtain a first target image; performing Fourier transform on each of the first target images to obtain a corresponding spectrum diagram, and performing statistics on the pixels in the spectrum diagram in units of rows to obtain the variance of the value range distribution to calibrate the row noise; subtracting the row noise from the first target image to obtain a second target image; and testing the second target image based on the Tukey-lambda distribution family to obtain a shape parameter different from, and statistically substituting the pixel values ​​of the second target image into the shape parameter to obtain the variance of the calibrated read noise.

[0013] In one embodiment of the present application, according to the corresponding relationship, noise is added to the noise-free image according to the brightness of each pixel in the noise-free image to obtain the noisy image, including: determining the noise intensity corresponding to the brightness of each pixel in the noise-free image based on the calibrated photon scattering noise, and adding a first noise signal to the corresponding pixel based on the determined noise intensity to obtain a first noise image; adding a second noise signal to each pixel in the first noise image based on the calibrated dark current noise to obtain a second noise image; adding a third noise signal to each row of pixels in the second noise image in units of rows based on the calibrated row noise to obtain a third noise image; and adding a fourth noise signal randomly generated by calibrated shape parameters and variance to each pixel in the third noise image based on the calibrated read noise as the noisy image.

[0014] In one embodiment of the present application, the denoising neural network model is trained based on the data to obtain the trained denoising neural network model, including: obtaining a noise image of the noise-free image based on the noise model, and inputting the noise image and the corresponding noisy image into the denoising neural network model; performing feature extraction of different depths on the input image through the encoder of the denoising neural network model, and transmitting the features of different depths to the decoder of the denoising neural network model through the residual block of the denoising neural network model, denoising the noisy image by fusing the features transmitted by the residual block through the decoder, and outputting a denoised predicted image; determining the difference between the denoised predicted image and the corresponding noise-free image based on a loss function, and adjusting the parameters of the denoising neural network model based on the difference; and stopping the training in response to satisfying a preset convergence condition to obtain the trained denoising neural network model.

[0015] According to the second aspect of the present application, a single-frame denoising device for an HDR module based on deep learning is provided, comprising: an image acquisition module, configured to acquire a single-frame image captured by an HDR module; and a denoising processing module, configured to input the single-frame image into a trained denoising neural network model, the trained denoising neural network model performs denoising on the single-frame image and outputs the denoised single-frame image; wherein the trained denoising neural network model is trained based on a data pair consisting of a noisy image and a corresponding noise-free image, and the noisy image is obtained by adding noise to the noise-free image based on the brightness of each pixel based on a calibrated noise model of the HDR module.

[0016] In one embodiment of the present application, it also includes: a noise calibration module, which is configured to calibrate the noise model of the HDR module based on multiple frames of original images of the calibration plate under different lighting environments and multiple frames of pure black images under a pure black environment collected by the HDR module to obtain a correspondence between different pixel brightness and noise intensity; a data generation module, which is configured to add noise to the noise-free image according to the brightness of each pixel in the noise-free image according to the correspondence to obtain the noisy image, and form a data pair with the corresponding noisy image and the noise-free image; and a model training module, which is configured to train the denoising neural network model based on the data pair to obtain the trained denoising neural network model.

[0017] In one embodiment of the present application, the noise calibration module includes: an additive noise calibration unit, configured to calibrate the additive noise in the noise model based on the multiple frames of pure black images; and a multiplicative noise calibration unit, configured to calibrate the multiplicative noise in the noise model based on the multiple frames of pure black images and the multiple frames of original images.

[0018] In one embodiment of the present application, the additive noise includes: at least one of read noise, dark current noise and row noise; and / or the multiplicative noise includes photon scattering noise.

[0019] In one embodiment of the present application, the multiplicative noise calibration unit includes: a photon scattering noise calibration unit, configured to determine the mean of each pixel for the multiple frames of pure black images, and determine the position of abnormal pixels based on a preset brightness threshold; subtract the additive noise from the multiple frames of original images to obtain a third target image, and determine the mean and variance of each pixel for multiple frames of the third target images under each lighting environment; determine the area to be detected in multiple frames of the third target images based on the position of the calibration plate in the image, and obtain the detection mask of multiple frames of the third target images based on the position of the abnormal pixels; and determine the target pixels in multiple frames of the third target images under different lighting environments based on the detection mask, count the mean and variance of each pixel in the target pixels, and obtain the corresponding relationship between different pixel brightness and noise intensity to calibrate the photon scattering noise.

[0020] In one embodiment of the present application, the correspondence between the brightness of different pixels and the noise intensity presents a multi-segment linear relationship.

[0021] In one embodiment of the present application, the multiplicative noise calibration unit also includes: a post-processing unit, configured to smooth the multiple linear relationships based on local filtering and local weighted linear regression; and determine the inflection points in the multiple linear relationships after smoothing based on a preset regional threshold, and perform linear fitting based on the inflection points to obtain the noise model corresponding to each linear relationship.

[0022] In one embodiment of the present application, the additive noise calibration unit includes: a dark current noise calibration unit, configured to determine the mean value of each pixel in the multiple frames of pure black images to calibrate the dark current noise, and take the average of the means of all pixels to obtain a uniform black level; a row noise calibration unit, configured to subtract the dark current noise from the multiple frames of pure black images and add the uniform black level to obtain a first target image; perform Fourier transform on each of the first target images to obtain a corresponding spectrum diagram, and perform statistics on the pixels in the spectrum diagram in units of rows to obtain the variance of the value range distribution to calibrate the row noise; a read noise calibration unit, configured to subtract the row noise from the first target image to obtain a second target image; test the second target image based on the Tukey-lambda distribution family to obtain a shape parameter different from, and statistically substitute the pixel values ​​of the second target image into the shape parameter to obtain the variance of the calibrated read noise.

[0023] In one embodiment of the present application, the data generation module includes: a photon scattering noise adding unit, configured to determine the noise intensity corresponding to the brightness of each pixel in the noise-free image based on the calibrated photon scattering noise, and add a first noise signal to the corresponding pixel based on the determined noise intensity to obtain a first noise image; a dark current noise adding unit, configured to add a second noise signal to each pixel in the first noise image based on the calibrated dark current noise to obtain a second noise image; a row noise adding unit, configured to add a third noise signal to each pixel in the second noise image in each row direction in units of rows based on the calibrated row noise to obtain a third noise image; and a read noise adding unit, configured to add a fourth noise signal randomly generated by calibrated shape parameters and variance to each pixel in the third noise image based on the calibrated read noise as the noisy image.

[0024] In one embodiment of the present application, the model training module includes: a pre-processing unit, configured to obtain a noise image of the noise-free image based on the noise model, and input the noise image and the corresponding noisy image into the denoising neural network model; a model training unit, configured to extract features of different depths on the input image through the encoder of the denoising neural network model, and transfer the features of different depths to the decoder of the denoising neural network model through the residual block of the denoising neural network model, and denoise the noisy image by fusing the features transferred by the residual block through the decoder, and output a denoised predicted image; determine the difference between the denoised predicted image and the corresponding noise-free image based on the loss function, and adjust the parameters of the denoising neural network model based on the difference; stop training in response to meeting a preset convergence condition, and obtain the trained denoising neural network model.

[0025] The electronic device provided according to the third aspect of the present application may include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the deep learning-based HDR module single-frame noise reduction method described in the first aspect of the present application.

[0026] The computer-readable storage medium provided according to the fourth aspect of the present application stores a computer program, and when the computer program is executed by a processor, the HDR module single-frame noise reduction method based on deep learning described in the first aspect of the present application is implemented.

[0027] According to the deep learning-based single-frame denoising method, device and equipment of the HDR module of the implementation mode of the present application, by calibrating the noise model of the HDR module, and adding noise based on the brightness of each pixel according to the calibrated noise model to construct an image simulated by the actual shooting of the HDR module, the similarity between the generated simulated image and the actual image shot by the HDR module can be ensured, and the denoising neural network model is trained by the simulated image to obtain a trained denoising neural network model. Compared with constructing training data through actual shooting, the volume of training data can be increased, thereby improving the adaptability of the trained denoising neural network model to different scenes. The single-frame image shot by the HDR module is denoised by the trained denoising neural network model to obtain a denoised single-frame image. Compared with the method of fitting the noise of the HDR module to additive Gaussian distribution noise, the denoising performance of the trained denoising neural network model can be improved, so that the obtained denoised single-frame image has a cleaner picture.

[0028] The contents described in this section are not intended to identify the key or important features of the embodiments of the present application, nor are they intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Other features, purposes and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present application. Among them:

[0030] Figure 1 It is a flowchart of a single-frame noise reduction method of an HDR module based on deep learning according to an implementation method of the present application;

[0031] Figure 2 It is a schematic diagram of a process of obtaining a trained denoising neural network model according to an implementation method of the present application;

[0032] Figure 3 is a schematic diagram of a process for collecting calibration data according to the method of the present application;

[0033] Figure 4 is a schematic diagram of a process for calibrating a noise model according to the method of the present application;

[0034] Figure 5 is a schematic diagram of a process for calibrating additive noise according to the method of the present application;

[0035] Figure 6 is a schematic diagram of a process for calibrating multiplicative noise according to the method of the present application;

[0036] Figure 7 is a schematic diagram of calibrating photon scattering noise according to the method of the present application;

[0037] Figure 8 is a schematic diagram of a detection mask obtained according to the method of the present application;

[0038] Fig. 9 is a schematic diagram of a process for removing random disturbances in a photon scattering noise model according to the method of the present application;

[0039] Fig.10 is a schematic diagram of a photon scattering noise model calibrated according to the method of the present application;

[0040] Fig.11 is a schematic diagram of a process of adding noise to a noise-free image according to the method of the present application;

[0041] Fig.12 is a schematic diagram of a process for training a neural network model according to the method of the present application;

[0042] Fig.13 A block diagram of a single-frame noise reduction device for an HDR module based on deep learning according to an embodiment of the present application;

[0043] Fig.14 A block diagram of a single-frame noise reduction device for an HDR module based on deep learning according to another embodiment of the present application;

[0044] Fig.15 It is a block diagram of an electronic device used to implement the deep learning-based HDR module single-frame noise reduction method of the implementation mode of the present application. DETAILED DESCRIPTION

[0045] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0046] HDR images usually have a wide brightness range and can capture a lot of information from shadow details to highlight details. However, due to the influence of image sensor noise and other environmental factors, HDR images are usually interfered by noise, resulting in reduced image quality and details.

[0047] Existing single-frame image denoising methods mainly include filtering based on spatial domain and filtering based on frequency domain.

[0048] Spatial domain-based filtering is a common image denoising method that operates at the pixel level of the image and estimates and adjusts the pixel value based on the neighborhood information around the pixel. It can usually be expressed in the form of formula 1:

[0049] Idenoised(x,y)=f(Inosy(x,y),Neighbor(x,y)) (Formula 1)

[0050] Among them, Idenoised(x,y) represents the pixel value after denoising at the coordinate (x,y); Inoisy(x,y) represents the original noisy pixel value at the coordinate (x,y); Neighbor(x,y) represents the neighborhood information around the pixel at the coordinate (x,y), which can be a fixed-size window or an adaptive local neighborhood; f(·) represents the filter function, which represents different spatial domain filtering methods, the more common ones are mean filtering and Gaussian filtering.

[0051] Filtering methods based on the spatial domain usually have the following disadvantages:

[0052] 1. Loss of details: Since spatial domain filtering usually adjusts pixel values ​​based on local neighborhood information, it may cause loss or blurring of image details. For images with complex noise patterns or high noise levels, spatial domain filtering methods often have difficulty in reducing noise and keeping details clear at the same time.

[0053] 2. Limited flexibility: Spatial domain filtering methods often have fixed filtering templates or parameter settings and cannot flexibly adapt to different types of noise or image scenes. This may lead to poor noise reduction effects in certain situations.

[0054] 3. Limited noise processing: Spatial domain filtering methods may be less effective in processing nonlinear noise or special noise. Some complex noise types, such as background noise or color noise, may not be well processed by existing spatial domain filtering methods.

[0055] Frequency domain filtering is a denoising method that converts an image from the spatial domain to the frequency domain for processing, and then converts the image from the frequency domain to the spatial domain. The usual processing flow is:

[0056] 1. Fourier transform: Frequency domain-based filtering first performs Fourier transform on the image to convert it to the frequency domain. Fourier transform can represent the image as the sum of frequency components, making the frequency domain filtering operation more convenient and efficient.

[0057] 2. Frequency domain filtering: In the frequency domain, images can be filtered, usually using filters or masks (also called frequency domain filter kernels) to adjust the amplitude and phase of frequency components. Common frequency domain filtering methods include: low-pass filters and high-pass filters. Low-pass filters reduce noise in images by suppressing high-frequency components, and high-pass filters enhance image details and edges by suppressing low-frequency components.

[0058] 3. After filtering in the frequency domain, the image needs to be converted from the frequency domain back to the spatial domain, and the filtered frequency domain image is converted into a spatial domain image using an inverse Fourier transform.

[0059] However, frequency domain based filtering methods usually reduce noise by suppressing high frequency components, but high frequency components contain edge and detail information. When high frequency components are suppressed or truncated, edges and details may be blurred or lost, resulting in unclear or unsmooth edges of the image.

[0060] Furthermore, the filtering method based on the frequency domain and the operations such as Fourier transform and inverse Fourier transform require a lot of computing resources and a long processing time. Especially for high-resolution images or large image sequences, the computational complexity of frequency domain filtering will be higher.

[0061] There is a linear relationship between the brightness and noise of an image generated by a normal image sensor. Simply put, as the number of photons captured by the image sensor increases, the noise generated will also increase accordingly, and this relationship is largely linear.

[0062] However, in order to capture the details from the brightest to the darkest in the real world and ensure that both dark and bright areas can retain details, the HDR module usually uses multiple images with different exposures and then fuses these images into an HDR image. Since the noise characteristics of the image sensor will be different under different exposure conditions, discontinuous noise changes may occur during the fusion process, especially at the junction of the bright and dark areas, that is, the fusion point. Since this involves the fusion of two images with different exposure characteristics and noise characteristics, the noise performance will show a "cliff-like" change. In this case, the noise of the HDR image is no longer just a simple linear relationship with the brightness, but is affected by multiple factors. Therefore, the denoising of the HDR image needs to take into account the brightness information in the image and perform corresponding processing according to the brightness level of different areas. However, the existing single-frame image denoising methods are difficult to deal with nonlinear, highly complex noise and detail loss caused by denoising.

[0063] In order to solve the above problems, an embodiment of the present application provides a single-frame noise reduction method 1000 for an HDR module based on deep learning.

[0064] Figure 1 FIG. 1 is a flow chart of a single-frame noise reduction method 1000 for an HDR module based on deep learning according to an embodiment of the present application. Figure 1 As shown, the HDR module single-frame noise reduction method 1000 based on deep learning in the implementation manner of the present application may include the following steps:

[0065] S100, obtaining a single frame image captured by the HDR module.

[0066] S200, inputting the single-frame image into a trained denoising neural network model, the trained denoising neural network model performs denoising on the single-frame image and outputs the denoised single-frame image; wherein the trained denoising neural network model is trained based on a data pair consisting of a noisy image and a corresponding noise-free image, and the noisy image is obtained by adding noise to the noise-free image based on the brightness of each pixel based on a noise model of a calibrated HDR module.

[0067] The method 1000 of the embodiment of the present application can be used to perform noise reduction processing on the image captured by the HDR module, by obtaining a single frame image in the image captured by the HDR module, inputting the obtained single frame image into a trained noise reduction neural network model, performing noise reduction processing on the single frame image through the trained noise reduction neural network model and outputting the noise-reduced single frame image. Among them, the trained noise reduction neural network model is obtained by training the noise reduction neural network model with data composed of a noisy image and a noise-free image, and the noisy image is obtained by calibrating the noise model of the HDR module, and adding noise to each pixel in the noise-free image according to the brightness of the pixel according to the calibrated noise model.

[0068] The implementation method of the present application provides a deep learning-based single-frame denoising method 1000 for an HDR module. By calibrating the noise model of the HDR module, noise is added based on the brightness of each pixel according to the calibrated noise model to construct an image simulated by the actual shooting of the HDR module. This ensures the similarity between the generated simulated image and the actual image shot by the HDR module. The denoising neural network model is trained by the simulated image to obtain a trained denoising neural network model. Compared with constructing training data through actual shooting, the volume of training data can be increased, thereby improving the adaptability of the trained denoising neural network model to different scenes. The single-frame image shot by the HDR module is denoised by the trained denoising neural network model to obtain a denoised single-frame image. Compared with the method of fitting the noise of the HDR module to additive Gaussian distribution noise, the denoising performance of the trained denoising neural network model can be improved, so that the obtained denoised single-frame image has a cleaner picture.

[0069] The HDR module single-frame denoising method 1000 based on deep learning provided by the embodiment of the present application has the advantages of more complete image texture preservation and better color restoration in terms of denoising ability compared with the existing blind denoising method, because there is a real noise-free high-definition image as a guide in the denoising neural network model training stage. At the same time, in an environment with a graphics card, a large number of calculations of the denoising neural network model can be processed in parallel, so it is superior to the existing blind denoising method in terms of time efficiency. The HDR module single-frame denoising method 1000 based on deep learning provided by the embodiment of the present application only needs to perform a denoising process on a single frame image of the HDR module that has completed the fusion, compared with the method of denoising and then fusing multiple images of different exposures of the HDR module, it can reduce the demand for computing resources and the time overhead of processing. At the same time, it can avoid the change of image structure that may be caused by denoising multiple images of different exposures, and thus increase the difficulty of subsequent image fusion.

[0070] It should be understood that the steps shown in method 1000 are not exclusive, and other steps may be performed before, after, or between any of the steps shown. In addition, some of the steps shown may be performed simultaneously, or may be performed in different steps. Figure 1 Executed in the order shown.

[0071] It should be understood that the HDR module in the embodiments of the present application may be an independent imaging device, such as a camera, a camcorder, etc., or it may be a photographic module, a video module, etc. used in products such as automobiles and mobile phones. The embodiments of the present application do not limit the form of the HDR module.

[0072] It should be understood that the image captured by the HDR module in the implementation mode of the present application may be a video including multiple frames of continuous images, or a single frame of photo images, and the implementation mode of the present application does not limit this. The single frame image to be processed may be each frame of the image captured by the HDR module, or a specific frame of the image captured by the HDR module, and the implementation mode of the present application does not limit this.

[0073] It should be noted that the noise-free image in the embodiments of the present application can be a clean high-definition image obtained by actual shooting, or can be a clean high-definition image downloaded from the Internet, such as an image in an existing training data set, etc. The embodiments of the present application do not limit the method for obtaining the noise-free image.

[0074] Furthermore, if Figure 2 As shown, Figure 2A schematic diagram of a process for obtaining a trained denoising neural network model according to an embodiment of the present application is shown. The HDR module single-frame denoising method 1000 based on deep learning according to an embodiment of the present application may also include the following steps:

[0075] S300, based on multiple frames of original images of the calibration plate in different lighting environments and multiple frames of pure black images in a pure black environment collected by the HDR module, calibrate the noise model of the HDR module to obtain the corresponding relationship between different pixel brightness and noise intensity.

[0076] S400, according to the corresponding relationship, adding noise to the noise-free image according to the brightness of each pixel in the noise-free image to obtain a noisy image, and forming a data pair with the corresponding noisy image and the noise-free image.

[0077] S500, training the denoising neural network model based on the data to obtain a trained denoising neural network model.

[0078] Before the method 1000 of the implementation mode of the present application performs noise reduction processing on the image captured by the HDR module, the image of a specific scene can be taken by the HDR module as calibration data. The specific scene can include different lighting environments and pure black environments, wherein the image of the calibration plate can be taken as calibration data under different lighting environments. Then, the noise model of the HDR module can be calibrated according to the calibration data to obtain the noise distribution of the HDR module at different brightnesses, wherein when calibrating the noise model, the noise can be decomposed into two parts, namely, signal-related noise and signal-independent noise, and calibrated separately. Afterwards, noise can be added to the noise-free image according to the brightness of each pixel according to the calibrated noise model to obtain a noisy image that simulates the noise distribution of the HDR module, and the corresponding noisy image and the noise-free image form a data pair. The denoising neural network model is trained through data pairs, wherein the noise in the noisy image in the data pair that is close to the real noise shape carries the prior information of multiple noises and is input into the denoising neural network model. The denoising neural network model is continuously fitted and the denoised prediction image is output. The noise-free image in the data pair is used as the target to calculate the loss function with the denoised prediction image, and finally the trained denoising neural network model is obtained after the denoising neural network model converges.

[0079] It should be understood that steps S100 to S200 and steps S300 to S500 shown in method 1000 can be executed by different devices, for example, steps S100 to S200 can be executed by a terminal, and steps S300 to S500 can be executed by a server; they can also be executed by the same device, for example, steps S100 to S200 and steps S300 to S500 can both be executed by a server.

[0080] It should be understood that steps S100 to S200 and steps S300 to S500 shown in method 1000 are not executed sequentially; after executing steps S300 to S500 to obtain the trained denoising neural network model, it is not necessary to continue to execute steps S100 to S200 to perform denoising processing on a single frame image; each time steps S100 to S200 are executed to perform denoising processing on a single frame image, it is only necessary to use the trained denoising neural network model, and it is not necessary to execute steps S300 to S500 to train the denoising neural network model.

[0081] Below Figure 2 The various steps of the HDR module single-frame noise reduction method 1000 based on deep learning are described in detail.

[0082] Step S300

[0083] Figure 3 The figure shows a schematic diagram of the process of collecting calibration data according to the method of the present application. Figure 3 As shown, before step S300, the following steps may also be included: S301, placing the 24-color card as a calibration plate in the center of the image as much as possible, and the size shall not exceed 1 / 3 of the entire image, and at the same time, the ambient light source shall be uniform, so that the calibration data can have higher quality; S302, adjusting the ambient light to lower and higher brightness, so that the pixel brightness value of the 24-color card area being photographed can cover the entire numerical range that the HDR module can express as much as possible under different lighting environments; S303, dividing the overall ambient light adjustment interval into several segments according to the lowest and highest lighting conditions that meet the conditions, which can be increased or decreased according to actual needs, and in each lighting environment, more than 60 frames of original images are captured by the HDR module; S304, placing the HDR module in a pure black environment to capture more than 60 frames of pure black images. Among them, the original image may refer to the original data captured by the image sensor without any image processing or compression, for example, the original image may be a Raw image in which the image sensor converts light information into a digital signal and stores it as a RAW format file to retain the maximum details and dynamic range of the image. A pure black image, namely a black frame, may be an image captured by placing the image sensor in a pure black environment without any light information.

[0084] According to the imaging principle, the noise model can usually include additive noise and multiplicative noise. First, when the image sensor receives photons, the photons are randomly scattered to adjacent pixels, causing the pixel values ​​to shift randomly and generate noise. Then after the image sensor completes signal acquisition, noise is generated in the signal readout stage due to random currents, circuit defects and other reasons. Therefore, the noise contained in the noise model can be divided into photon scattering noise and readout noise, where photon scattering noise is a multiplicative noise superimposed on the original signal, and readout noise is an additive noise superimposed on the original signal, for example, it can include: at least one of read noise, dark current noise and row noise. Optionally, the overall noise model can be expressed in the form of Formula 2:

[0085] N=NpI+Nread+Nd+Nr (Formula 2)

[0086] Wherein, N represents a noisy image; Np represents the intensity of photon scattering noise; I represents an ideal noise-free image; Nread represents the intensity of read noise; Nd represents the intensity of dark current noise; and Nr represents the intensity of row noise.

[0087] The actual noise distribution can be deduced from the image in reverse to the imaging principle. The additive noise can be calibrated first, and then the influence of the additive noise can be removed from the image to calibrate the multiplicative noise, so that the calibrated noise model is more accurate. Figure 4 As shown, step S300 may include the following steps: S310, calibrating additive noise in the noise model based on multiple frames of pure black images; S320, calibrating multiplicative noise in the noise model based on multiple frames of pure black images and multiple frames of original images.

[0088] In some optional implementations, the additive noise may include: read noise, dark current noise, and row noise. Figure 5 As shown, step S310 may include the following steps: S311, determining the mean value of each pixel in multiple frames of pure black images to calibrate the dark current noise, and averaging the means of all pixels to obtain a uniform black level; S312, subtracting the dark current noise from the multiple frames of pure black images and adding the uniform black level to obtain a first target image; S313, performing Fourier transform on each first target image to obtain a corresponding spectrum diagram, and performing statistics on the pixels in the spectrum diagram in units of rows to obtain the variance of the value range distribution to calibrate the row noise; S314, subtracting the row noise from the first target image to obtain a second target image; S315, testing the second target image based on the Tukey-lambda distribution family to obtain a shape parameter different from, and statistically substituting the pixel values ​​of the second target image into the shape parameter to obtain the variance of the calibrated read noise.

[0089] Among them, when calibrating the dark current noise, step S311 counts the average value of each pixel of all black frames to obtain a pixel-level black level different from the unified black level of the whole image, that is, the dark current noise Nd, and at the same time takes the average value of all pixels and rounds to integers to obtain an approximate unified black level of the whole image. When calibrating the row noise, steps S312 and S313 subtract the pixel-level black level from the black frame and add the unified black level of the whole image to obtain an image of N-Nd, and then perform Fourier transform on the obtained image to obtain a spectrum diagram. The spectrum diagram can reveal the row noise existing in the black frame in the horizontal and vertical directions. The row noise is caused by the problem of inconsistent amplification current between rows when the signal is usually read out by row. By counting the pixels in the spectrum diagram in units of rows, the variance of the value range distribution can be obtained, so as to calibrate the row noise Nr. When calibrating the read noise, steps S314 and S315 further subtract the influence of the row noise to obtain the N-Nd-Nr image, and then perform a Tukey-lambda distribution family test on the obtained image to obtain a shape parameter λ that is different from the Gaussian distribution. By loading the obtained image, counting the pixel values, and substituting the shape parameter λ, the variance of the read noise Nread can be obtained.

[0090] In some other optional embodiments, the multiplicative noise may include photon scattering noise. Figure 6 As shown, step S320 may include the following steps: S321, determining the mean of each pixel for multiple frames of pure black images, and determining the position of abnormal pixels based on a preset brightness threshold; S322, subtracting additive noise from multiple frames of original images to obtain a third target image, and determining the mean and variance of each pixel for multiple frames of third target images under each lighting environment; S323, determining the area to be detected in multiple frames of third target images based on the position of the calibration plate in the image, and obtaining a detection mask for multiple frames of third target images based on the position of the abnormal pixels; S324, determining the target pixels in multiple frames of third target images under different lighting environments based on the detection mask, counting the mean and variance of each pixel in the target pixels, and obtaining the corresponding relationship between different pixel brightness and noise intensity to calibrate the photon scattering noise.

[0091] like Figure 7As shown, when calibrating the photon scattering noise, step S321 averages the pure black image obtained in the pure black environment, obtains a set of multi-frame average data with a length and width as the image size, sets a reasonable brightness threshold, for example, in 12-bit data, the upper limit can be 300, the lower limit can be 10, and the abnormal points outside the brightness threshold are recorded. Step S322 subtracts the additive noise calibrated in step S310 from all images taken under each lighting environment to obtain the NpI image, averages the images obtained from multiple frames, calculates the variance of each pixel position under multiple frames, and normalizes them. Step S323 roughly delineates the area according to the position of the 24-color card used as the calibration plate in the obtained image. For example, edge detection operators such as Sobel and Canny can be used to obtain the edge area of ​​the image, and certain dilation and corrosion operations can be performed according to the actual image performance. Combined with the position of the abnormal point obtained in step S321 and the above-mentioned delineated area, the pixels that do not need to be counted in the obtained image are excluded, and finally the following is obtained. Figure 8 The detection mask shown, Figure 8 The pixels in the black part will be used to count the noise. Step S324 introduces the mean and variance of each pixel of the whole image counted under each lighting environment, and superimposes the detection mask obtained in step S323, removes the uncounted part of the mask, and counts the brightness value of the remaining pixels after screening according to their positions and the multi-frame mean of the pixels at their positions, as well as the corresponding variance. In this way, the variance corresponding to each brightness value under a certain specific lighting condition can be obtained. In order to more comprehensively analyze the noise intensity, the data of the images obtained under all different lighting conditions are finally merged. This means that for the same brightness value, multiple variance values ​​can be obtained from multiple images, and a more accurate and stable data can be obtained by calculating the mean of these variance values. After completing the above steps, a noise model that presents a multi-segment linear relationship as a whole will be obtained.

[0092] Optionally, the noise model obtained in steps S321 to S324 usually presents slight disturbances locally. These slight disturbances are due to the fact that noise statistics are essentially statistics of random deviations, and the noise model should accept random errors with a small disturbance range. Fig. 9 As shown, step S300 may also include the following steps: S325, smoothing multiple linear relationships based on local filtering and local weighted linear regression; S326, determining the inflection points in the multiple linear relationships after smoothing based on a preset regional threshold, and performing linear fitting based on the inflection points to obtain a noise model corresponding to each linear relationship.

[0093] After step S324, the random disturbance in the noise model initially obtained in step S324 can be corrected through steps S325 and S326 to obtain a global noise model that can be described by a mathematical model. Step S325 can eliminate random disturbances by filtering locally, mainly by combining moving average filtering with low-pass filtering, and in larger local areas, local weighted linear regression can be used to make the line segments smoother, thereby obtaining a model that is easy to be counted. Step S326 calculates the first-order and second-order derivatives of the above-obtained model to obtain the inflection points in the model. By setting regional thresholds, the model is prevented from frequently changing positively and negatively in local areas, resulting in overfitting problems. These inflection points are counted, and linear fitting is used in turn to obtain the noise model of each linear relationship. The final noise model is as follows: Fig.10 shown.

[0094] Step S400

[0095] like Fig.11 As shown, in step S400, according to the corresponding relationship, noise is added to the noise-free image according to the brightness of each pixel in the noise-free image to obtain a noisy image, which may include the following steps: S410, determining the noise intensity corresponding to the brightness of each pixel in the noise-free image based on the calibrated photon scattering noise, and adding a first noise signal to the corresponding pixel based on the determined noise intensity to obtain a first noise image; S420, adding a second noise signal to each pixel in the first noise image based on the calibrated dark current noise to obtain a second noise image; S430, adding a third noise signal to each pixel in each row direction in the second noise image in units of rows based on the calibrated row noise to obtain a third noise image; S440, adding a fourth noise signal randomly generated by calibrated shape parameters and variance to each pixel in the third noise image based on the calibrated read noise as a noisy image.

[0096] Noise can be added to each high-definition image I used for training in sequence according to the calibrated noise model and the imaging process. Among them, step S410 adds photon scattering noise, and a random noise signal that conforms to the Poisson distribution can be generated according to the noise intensity σp corresponding to the image pixel. At this time, the image can be represented as N=I+N(0,σp), where N(·) represents the Gaussian distribution. Step S420 adds dark current noise, adding a given value to each pixel in the image obtained in the previous step. Step S430 adds row noise, and a value that conforms to the Gaussian distribution can be randomly added to each row of data on the basis of the original signal under a given variance in row units. S440 adds read noise, and a heterogeneous long-tailed Gaussian distribution can be used. On the basis of a mean of 0, a calibrated shape parameter and a randomly generated value of the variance are added to each pixel to obtain an image with random noise similar to the actual shooting effect.

[0097] Step S500

[0098] like Fig.12 As shown, in step S500, the denoising neural network model is trained based on the data to obtain a trained denoising neural network model, which may include the following steps: S510, a noise image of the noise-free image is obtained based on the noise model, and the noise image and the corresponding noisy image are input into the denoising neural network model; S520, features of different depths are extracted from the input image through the encoder of the denoising neural network model, and the features of different depths are transferred to the decoder of the denoising neural network model through the residual block of the denoising neural network model, and the noisy image is denoised by fusing the features transferred by the residual block through the decoder, and the denoising is output. The predicted image after denoising is output; S530, the difference between the predicted image after denoising and the corresponding noise-free image is determined based on the loss function, and the parameters of the denoising neural network model are adjusted based on the difference; S540, the training is stopped in response to meeting the preset convergence condition to obtain a trained denoising neural network model.

[0099] A deep residual network model including an encoder and a decoder is used as a denoising neural network model, such as a U-Net network, which can capture global context information in the image and restore details in the decoder, which helps to extract useful features in the image and effectively reduce noise. At the same time, the residual block jump connection mechanism helps to retain detail information between different levels. This connection mode combines low-level features with high-level features, thereby retaining more detail information in the denoising process. Among them, step S510 preprocesses the noisy image and its corresponding noise-free image. Considering that when the trained denoising neural network model is formally deployed, the input image is a noisy image, and there is no high-definition noise-free image as prior information, it is necessary to simulate this situation in the training stage. The black level is subtracted from the noisy image to restore the more realistic electronic signal of the image, and then the pixel variance under the influence of the noise signal is obtained according to the calibrated noise model, and the variance of the additive noise full image is added to obtain a std_map with a resolution consistent with the image. The std_map reveals the approximate noise intensity of each pixel point, thereby guiding the network to judge the denoising intensity. The std_map is used as an information layer together with the noisy image to input the denoising neural network model for training. By introducing the noisy image std_map as an information layer, it helps the network to better distinguish different noise intensities, and can denoise different areas with different strengths, which can effectively alleviate the smearing problem caused by over-noise.

[0100] Step S520: The convolution layer of the denoising neural network model will gradually decompose the image into feature layers of different sizes, extract features of different depths through residual blocks, and pass the feature information to a deeper network to enhance the network's ability to represent features. Then, this information is decoded again into a denoised residual layer through deconvolution decoding, and the residual layer is fused with the original noisy image to obtain a denoised predicted image. Step S503: The selection of the loss function may vary according to the task. For example, considering that high-frequency details are easily erased along with the noise in denoising, and color noise causes discoloration after denoising, etc., a color deviation loss function and a structural loss function may be used. The loss function calculates the difference between the predicted image output by the denoising neural network model and the high-definition noise-free image by comparing the minimum absolute value deviation between the two, and adjusts the parameters of the denoising neural network model. By continuously iteratively adjusting the network parameters and calculating the loss function, the loss function is finally converged. Step S540: When the loss function converges, the required trained denoising neural network model can be obtained.

[0101] The embodiments of the present application also provide a single-frame noise reduction device 2000 for an HDR module based on deep learning.

[0102] Fig.13 The structure diagram of the HDR module single frame noise reduction device 2000 based on deep learning according to an embodiment of the present application is shown. The HDR module single frame noise reduction device 2000 based on deep learning according to the embodiment of the present application can be applied to the HDR module single frame noise reduction device 1000 based on deep learning according to any of the above embodiments. Fig.13 As shown, the HDR module single-frame noise reduction device 2000 based on deep learning in the implementation manner of the present application may include:

[0103] The image acquisition module 110 is configured to acquire a single frame image captured by the HDR module; and

[0104] The denoising processing module 120 is configured to input the single-frame image into a trained denoising neural network model, and the trained denoising neural network model performs denoising on the single-frame image and outputs the denoised single-frame image; wherein the trained denoising neural network model is trained based on a data pair consisting of a noisy image and a corresponding noise-free image, and the noisy image is obtained by adding noise to the noise-free image based on the brightness of each pixel based on the calibrated noise model of the HDR module.

[0105] Alternatively, if Fig.14 As shown, Fig.14A schematic diagram of the structure of a HDR module single frame noise reduction device 2000 based on deep learning according to another embodiment of the present application is shown. The HDR module single frame noise reduction device 2000 based on deep learning according to the embodiment of the present application may also include:

[0106] The noise calibration module 130 is configured to calibrate the noise model of the HDR module based on multiple frames of original images of the calibration plate under different lighting environments and multiple frames of pure black images under a pure black environment collected by the HDR module to obtain a corresponding relationship between different pixel brightness and noise intensity;

[0107] The data generating module 140 is configured to add noise to the noise-free image according to the corresponding relationship and the brightness of each pixel in the noise-free image to obtain the noisy image, and form a data pair with the corresponding noisy image and the noise-free image; and

[0108] The model training module 150 is configured to train the denoising neural network model based on the data to obtain the trained denoising neural network model.

[0109] Optionally, the noise calibration module 130 includes:

[0110] an additive noise calibration unit, configured to calibrate the additive noise in the noise model based on the multiple frames of pure black images; and

[0111] The multiplicative noise calibration unit is configured to calibrate the multiplicative noise in the noise model based on the multiple frames of pure black images and the multiple frames of original images.

[0112] Optionally, the additive noise includes: at least one of read noise, dark current noise and row noise; and / or the multiplicative noise includes photon scattering noise.

[0113] Optionally, the multiplicative noise calibration unit comprises:

[0114] The photon scattering noise calibration unit is configured to determine the mean value of each pixel for the multiple frames of pure black images, and determine the position of abnormal pixels based on a preset brightness threshold; subtract the additive noise from the multiple frames of original images to obtain a third target image, and determine the mean value and variance of each pixel for the multiple frames of the third target images under each lighting environment; determine the area to be detected in the multiple frames of the third target images based on the position of the calibration plate in the image, and obtain the detection mask of the multiple frames of the third target images based on the position of the abnormal pixels; and determine the target pixels in the multiple frames of the third target images under different lighting environments based on the detection mask, count the mean value and variance of each pixel in the target pixels, and obtain the corresponding relationship between different pixel brightness and noise intensity to calibrate the photon scattering noise.

[0115] Optionally, the corresponding relationship between the brightness of different pixels and the noise intensity presents a multi-segment linear relationship.

[0116] Optionally, the multiplicative noise calibration unit further includes:

[0117] The post-processing unit is configured to smooth the multiple linear relationships based on local filtering and local weighted linear regression; and determine the inflection points in the multiple linear relationships after smoothing based on a preset regional threshold, and perform linear fitting based on the inflection points to obtain the noise model corresponding to each linear relationship.

[0118] Optionally, the additive noise calibration unit comprises:

[0119] A dark current noise calibration unit is configured to determine the mean value of each pixel in the plurality of frames of pure black images to calibrate the dark current noise, and average the mean values ​​of all pixels to obtain a uniform black level;

[0120] A row noise calibration unit is configured to subtract dark current noise from the plurality of frames of pure black images and add the uniform black level to obtain a first target image; perform Fourier transform on each of the first target images to obtain a corresponding spectrum diagram, and perform statistics on pixels in the spectrum diagram in units of rows to obtain a variance of a value range distribution to calibrate the row noise;

[0121] The read noise calibration unit is configured to subtract the row noise from the first target image to obtain a second target image; test the second target image based on the Tukey-lambda distribution family to obtain a shape parameter different from , and count the pixel values ​​of the second target image and substitute them into the shape parameter to obtain the variance of the calibrated read noise.

[0122] Optionally, the data generation module 140 includes:

[0123] a photon scattering noise adding unit, configured to determine a noise intensity corresponding to the brightness of each pixel in the noise-free image based on the calibrated photon scattering noise, and add a first noise signal to the corresponding pixel based on the determined noise intensity to obtain a first noise image;

[0124] a dark current noise adding unit, configured to add a second noise signal to each pixel in the first noise image based on the calibrated dark current noise to obtain a second noise image;

[0125] a row noise adding unit configured to add a third noise signal to each row of pixels in the second noise image in units of rows based on the calibrated row noise to obtain a third noise image; and

[0126] The read noise adding unit is configured to add a fourth noise signal randomly generated with a calibrated shape parameter and variance to each pixel in the third noise image based on the calibrated read noise as the noisy image.

[0127] Optionally, the model training module 150 includes:

[0128] A pre-processing unit is configured to obtain a noise image of the noise-free image based on the noise model, and input the noise image and the corresponding noisy image into the denoising neural network model;

[0129] The model training unit is configured to extract features of different depths on the input image through the encoder of the denoising neural network model, and transfer the features of different depths to the decoder of the denoising neural network model through the residual block of the denoising neural network model, and denoise the noisy image by fusing the features transferred by the residual block through the decoder, and output a denoised predicted image; determine the difference between the denoised predicted image and the corresponding noise-free image based on the loss function, and adjust the parameters of the denoising neural network model based on the difference; stop training in response to meeting a preset convergence condition, and obtain the trained denoising neural network model.

[0130] In addition, an embodiment of the present application also provides an electronic device, which includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the above-mentioned deep learning-based HDR module single-frame noise reduction method 1000.

[0131] The embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned deep learning-based HDR module single-frame noise reduction method 1000.

[0132] Fig.15 A schematic block diagram of an example electronic device 3000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers. The electronic device can also represent various forms of mobile devices that can run computing programs. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0133] like Fig.15As shown, the electronic device 3000 includes a computing unit 201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 202 or a computer program loaded from a storage unit 208 into a random access memory (RAM) 203. In the RAM 203, various programs and data required for the operation of the electronic device 3000 can also be stored. The computing unit 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0134] A number of components in the electronic device 3000 are connected to the I / O interface 205, including: an input unit 206, such as a touch screen, etc.; an output unit 207, such as various types of displays, speakers, etc.; a storage unit 208, such as a disk, etc.; and a communication unit 209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 209 allows the electronic device 3000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0135] The computing unit 201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 201 performs the various methods and processes described above, such as the HDR module single-frame denoising method 1000 based on deep learning. For example, in some embodiments, the HDR module single-frame denoising method 1000 based on deep learning may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 208. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 3000 via the ROM 202 and / or the communication unit 209. When the computer program is loaded into the RAM 203 and executed by the computing unit 201, one or more steps of the HDR module single-frame denoising method 1000 based on deep learning described above may be performed. Alternatively, in other embodiments, the computing unit 201 may be configured to execute the deep learning-based HDR module single-frame noise reduction method 1000 in any other appropriate manner (e.g., by means of firmware).

[0136] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow diagram and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0138] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0139] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0140] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A single-frame noise reduction method for an HDR module based on deep learning, characterized in that: include: Get a single frame image captured by the HDR module; as well as The single-frame image is input into a trained denoising neural network model, and the trained denoising neural network model performs denoising on the single-frame image and outputs the denoised single-frame image; wherein the trained denoising neural network model is trained based on a data pair consisting of a noisy image and a corresponding noise-free image, and the noisy image is obtained by adding noise to the noise-free image based on the brightness of each pixel based on the calibrated noise model of the HDR module.

2. The method according to claim 1, characterized in that Also includes: Based on multiple frames of original images of the calibration plate in different lighting environments and multiple frames of pure black images in a pure black environment collected by the HDR module, the noise model of the HDR module is calibrated to obtain a corresponding relationship between different pixel brightness and noise intensity; According to the corresponding relationship, noise is added to the noise-free image according to the brightness of each pixel in the noise-free image to obtain the noisy image, and the corresponding noisy image and the noise-free image form a data pair; as well as The denoising neural network model is trained based on the data to obtain the trained denoising neural network model.

3. The method according to claim 2, characterized in that Based on multiple frames of original images of the calibration plate in different lighting environments and multiple frames of pure black images in a pure black environment collected by the HDR module, the noise model of the HDR module is calibrated to obtain the corresponding relationship between different pixel brightness and noise intensity, including: Calibrate the additive noise in the noise model based on the multiple frames of pure black images; and The multiplicative noise in the noise model is calibrated based on the multiple frames of pure black images and the multiple frames of original images.

4. The method according to claim 3, characterized in that The additive noise includes: at least one of read noise, dark current noise and row noise; and / or, The multiplicative noise includes photon scattering noise.

5. The method according to claim 4, characterized in that Calibrating the multiplicative noise in the noise model based on the multiple frames of pure black images and the multiple frames of original images includes: Determine the average value of each pixel of the plurality of frames of pure black images, and determine the position of abnormal pixels based on a preset brightness threshold; Subtracting the additive noise from the multiple frames of original images to obtain a third target image, and determining the mean and variance of each pixel for the multiple frames of the third target image under each lighting environment; Determine the area to be detected in multiple frames of the third target image based on the position of the calibration plate in the image, and obtain the detection mask of the multiple frames of the third target image based on the position of the abnormal pixel; and The target pixels in multiple frames of the third target image under different lighting environments are determined based on the detection mask, the mean and variance of each pixel in the target pixels are counted, and the corresponding relationship between different pixel brightness and noise intensity is obtained to calibrate the photon scattering noise.

6. The method according to claim 5, characterized in that The corresponding relationship between the brightness of different pixels and the noise intensity presents a multi-segment linear relationship.

7. The method according to claim 6, characterized in that Calibrate the multiplicative noise in the noise model based on the multiple frames of pure black images and the multiple frames of original images, further comprising: Smoothing the multiple linear relationships based on local filtering and local weighted linear regression; and The inflection points in the multiple linear relationships after smoothing are determined based on a preset regional threshold, and a linear fitting is performed based on the inflection points to obtain a noise model corresponding to each linear relationship.

8. A single-frame noise reduction device for an HDR module based on deep learning, characterized in that: include: An image acquisition module is configured to acquire a single frame image captured by the HDR module; as well as The denoising processing module is configured to input the single-frame image into a trained denoising neural network model, and the trained denoising neural network model performs denoising on the single-frame image and outputs the denoised single-frame image; wherein the trained denoising neural network model is trained based on a data pair consisting of a noisy image and a corresponding noise-free image, and the noisy image is obtained by adding noise to the noise-free image based on the brightness of each pixel based on the calibrated noise model of the HDR module.

9. An electronic device, characterized in that: include: at least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the deep learning-based HDR module single-frame denoising method described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the HDR module single-frame noise reduction method based on deep learning as described in any one of claims 1 to 7 is implemented.

Images