Image noise reduction processing method, device, storage medium and electronic device

Abstract

The present disclosure provides an image denoising processing method, device, computer-readable storage medium and electronic device, and relates to the field of image and video processing technology. One of the image denoising processing methods includes: a second denoising module obtains a second image and first reference information output after the first denoising module performs denoising on the first image; the second denoising module performs denoising on the second image based on the first reference information, and outputs a denoised image corresponding to the first image; wherein the first denoising module and the second denoising module run on different processors respectively; or, one of the first denoising module and the second denoising module is a hardware denoising module, and the other is a software denoising module. The present disclosure can improve the image denoising effect.

Description

Background Art

[0002] Image noise reduction refers to the processing technology of reducing the noise in digital images. The sources of noise in images are usually multifaceted, including thermal noise and dark current noise of the image sensor during image acquisition, and signal noise caused by external interference during image transmission.

[0003] Currently, image noise reduction processing on electronic devices is mostly achieved through fixed methods such as filtering, but the image noise reduction effect that can be achieved is not ideal, resulting in low output image quality.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to those skilled in the art. Summary of the invention

[0005] The present disclosure provides an image noise reduction processing method, an image noise reduction processing device, a computer-readable storage medium and an electronic device, thereby improving the image noise reduction effect at least to a certain extent.

[0006] According to a first aspect of the present disclosure, there is provided an image denoising method, comprising: a second denoising module acquiring a second image and first reference information outputted by the first denoising module after denoising the first image; the second denoising module denoising the second image based on the first reference information, and outputting a denoised image corresponding to the first image; wherein the first denoising module and the second denoising module are respectively run on different processors; or, one of the first denoising module and the second denoising module is a hardware denoising module, and the other is a software denoising module.

[0007] According to a second aspect of the present disclosure, there is provided an image denoising method, comprising: a first denoising module acquires second reference information output by a second denoising module; the first denoising module performs denoising on a first image based on the second reference information, and outputs the obtained second image to the second denoising module, so that the second denoising module performs denoising on the second image and then outputs a denoised image corresponding to the first image; wherein the first denoising module and the second denoising module run on different processors respectively; or, one of the first denoising module and the second denoising module is a hardware denoising module, and the other is a software denoising module.

[0008] According to a third aspect of the present disclosure, there is provided an image denoising processing device, comprising: a first denoising module, configured to perform denoising processing on a first image and then output a second image and first reference information; a second denoising module, configured to obtain the second image and the first reference information, perform denoising processing on the second image based on the first reference information, and output a denoised image corresponding to the first image; wherein the first denoising module and the second denoising module run on different processors respectively; or, one of the first denoising module and the second denoising module is a hardware denoising module, and the other is a software denoising module.

[0009] According to a fourth aspect of the present disclosure, there is provided an image denoising processing device, comprising: a first denoising module, configured to obtain second reference information output by a second denoising module, perform denoising on a first image based on the second reference information, and output the obtained second image to the second denoising module; the second denoising module, configured to output the second reference information, and perform denoising on the second image and then output a denoised image corresponding to the first image; wherein the first denoising module and the second denoising module run on different processors respectively; or, one of the first denoising module and the second denoising module is a hardware denoising module, and the other is a software denoising module.

[0010] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image denoising processing method of the first or second aspect and its possible implementation methods are implemented.

[0011] According to a sixth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image denoising processing method of the first or second aspect and its possible implementation method by executing the executable instructions.

[0012] The technical solution disclosed in this disclosure has the following beneficial effects:

[0013] The first noise reduction module performs noise reduction processing on the first image, and outputs the obtained second image and the first reference information to the second noise reduction module. The second noise reduction module then performs noise reduction processing on the second image based on the first reference information, and outputs the noise-reduced image. Therefore, this solution communicates the first reference information between the first noise reduction module and the second noise reduction module, so that the second noise reduction module uses more appropriate configuration information to perform noise reduction processing based on the first reference information, thereby realizing the effective combination of the first noise reduction module and the second noise reduction module, which is conducive to improving the noise reduction effect, and reduces the repeated processing and excessive noise reduction that may be generated by the two noise reduction modules, reduces power consumption and memory consumption, and improves the problem of excessive loss of image detail information.

[0014] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0016] Figure 1 A schematic diagram showing the structure of an electronic device in this exemplary embodiment is shown;

[0017] Figure 2 A flowchart of an image noise reduction method in this exemplary embodiment is shown;

[0018] Figure 3 A schematic diagram showing a block difference image and a pixel difference image in this exemplary embodiment;

[0019] Figure 4 A flowchart showing the noise reduction process performed by the second noise reduction module in this exemplary embodiment is shown;

[0020] Figure 5 A schematic flowchart of an image noise reduction processing method in this exemplary embodiment is shown;

[0021] Figure 6 A flowchart showing another method for image noise reduction processing in this exemplary embodiment is shown;

[0022] Figure 7 A schematic diagram showing the structure of an image noise reduction processing device in this exemplary embodiment is shown;

[0023] Figure 8 A schematic structural diagram of another image noise reduction processing device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0024] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0025] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0026] The ISP (Image Signal Processor) of general electronic devices is equipped with a relevant noise reduction module, which can suppress image noise in the RAW domain (referring to the RAW format image data output by the image sensor) or other levels. After the ISP processing is completed, the image is output to the relevant application, and then the software noise reduction module performs further noise reduction on the image. In other words, the electronic device includes two noise reduction modules, which perform image noise reduction processing in sequence.

[0027] The inventors have found that there is a lack of effective combination between the two noise reduction modules. For example, the ISP noise reduction module performs noise reduction processing according to preset parameters, and the software noise reduction module performs noise reduction processing according to image features. The processing processes of the two are independent of each other, which is not conducive to improving the noise reduction effect. There may be repeated or nearly repeated steps in the algorithms executed by the two noise reduction modules, resulting in high power consumption and memory consumption of the noise reduction processing. In addition, the superposition of the two noise reduction modules is prone to excessive noise reduction, or there is excessive noise reduction in a single noise reduction module, resulting in excessive loss of image detail information.

[0028] In view of one or more of the above problems, an exemplary embodiment of the present disclosure provides an image noise reduction processing method. In order to implement the image noise reduction processing method, an exemplary embodiment of the present disclosure also provides an electronic device. The electronic device may be a smart phone, a tablet computer, a digital camera, a drone, a virtual reality / augmented reality device, etc. Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to implement the image noise reduction processing method by executing the executable instructions.

[0029] Below Figure 1 The mobile terminal 100 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 1 The construction in can also be applied to fixed type equipment.

[0030] like Figure 1 As shown, the mobile terminal 100 may specifically include: a processor 101, a memory 102, a bus 103, a mobile communication module 104, an antenna 1, a wireless communication module 105, an antenna 2, a display screen 106, a camera module 107, an audio module 108, a power module 109 and a sensor module 110.

[0031] The processor 101 generally refers to all processors in the mobile terminal 100. For example, the processor 101 may be in the form of a SoC (System on Chip) on which all processors with different functions are integrated. Exemplarily, the processor 101 may include a CPU (Central Processing Unit) 1011, a GPU (Graphics Processing Unit) 1012, and an ISP (Image Signal Processor) 1013. The CPU 1011 is responsible for running system programs and major application programs. In one embodiment, the CPU 1011 may further include an AP (Application Processor), a modem processor, a baseband processor, etc. The GPU 1012 is responsible for running large graphics or image processing programs. ISP 1013 is responsible for processing image signals. It can receive the original image data output by the camera module 107, perform black level compensation, noise reduction, demosaicing, automatic white balance, color correction and other processing, and output the processed image to CPU 1011 for use and further processing by the application running in CPU 1011.

[0032] In one implementation, the processor 101 may also include one or more of a controller, an encoder, a decoder, a DSP (Digital Signal Processor), and an NPU (Neural-Network Processing Unit).

[0033] The encoder can encode (i.e. compress) an image or video to reduce the data size for easy storage or transmission. The decoder can decode (i.e. decompress) the encoded data of the image or video to restore the image or video data. In this exemplary embodiment, the encoder and decoder can support images or videos in multiple encoding formats, such as: image formats such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap), MPEG (Moving Picture Experts Group) 1, MPEG2, H.263, H.264, HEVC (High Efficiency Video Coding), and other video formats.

[0034] In this exemplary embodiment, the image noise reduction processing method can be specifically executed by the above-mentioned CPU 1011, GPU 1012 or ISP 1013.

[0035] The processor 101 may be connected to the memory 102 or other components via a bus 103 .

[0036] The memory 102 may be used to store computer executable program codes, which include instructions. The processor 101 executes various functional applications and data processing of the mobile terminal 100 by running the instructions stored in the memory 102. The memory 102 may also store application data, such as images, videos and other files.

[0037] The communication function of the mobile terminal 100 can be realized by the mobile communication module 104, the antenna 1, the wireless communication module 105, the antenna 2, etc. The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 104 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 100. The wireless communication module 105 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near field communication applied to the mobile terminal 100.

[0038] The display screen 106 is used to implement display functions, such as displaying user interfaces, images, videos, etc.

[0039] The camera module 107 is used to implement shooting functions, such as shooting the current image, video, etc. Generally, the camera module 107 may include a lens and an image sensor. The lens is used to adjust the incident light and image formation. The image sensor is used to sense the incident light and convert the light signal into a digital signal. The digital signal may be the original image data, which is further transmitted to the ISP 1013 for processing. In one embodiment, the ISP 1013 may be disposed in the camera module 107.

[0040] The audio module 108 is used to implement audio functions, such as playing audio, collecting voice, etc.

[0041] The power module 109 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc.

[0042] The sensor module 110 may include a pressure sensor 1101, an inertial sensor 1102, an air pressure sensor 1103, etc., to implement corresponding sensing detection functions. For example, the inertial sensor 1102 may include an accelerometer, a gyroscope, etc., for outputting acceleration data, gyroscope data, etc., to characterize the motion and posture information of the mobile terminal 100.

[0043] Combine the following Figure 2 The image noise reduction processing method in this exemplary embodiment is described. Figure 2 An exemplary process of the image noise reduction processing method is shown, which may include:

[0044] Step S210, the second denoising module acquires the second image and the first reference information output after the first denoising module performs denoising on the first image;

[0045] Step S220: The second denoising module performs denoising on the second image based on the first reference information, and outputs a denoised image corresponding to the first image.

[0046] The first noise reduction module and the second noise reduction module run on different processors respectively, for example, the first noise reduction module runs on the ISP and the second noise reduction module runs on the CPU, or the first noise reduction module runs on the CPU and the second noise reduction module runs on the ISP. Alternatively, one of the first noise reduction module and the second noise reduction module is a hardware noise reduction module and the other is a software noise reduction module, for example, the first noise reduction module is a hardware noise reduction module of the ISP, which performs image noise reduction processing at the hardware level, and the second noise reduction module is a software noise reduction module, which performs image noise reduction processing at the software application level, or the first noise reduction module is a software noise reduction module and the second noise reduction module is a hardware noise reduction module of the ISP.

[0047] Based on the above method, the first noise reduction module performs noise reduction processing on the first image, outputs the obtained second image and the first reference information to the second noise reduction module, and the second noise reduction module performs noise reduction processing on the second image based on the first reference information, and outputs the noise-reduced image. Therefore, this solution communicates the first reference information between the first noise reduction module and the second noise reduction module, so that the second noise reduction module uses more appropriate configuration information for noise reduction processing based on the first reference information, thereby realizing the effective combination of the first noise reduction module and the second noise reduction module, which is conducive to improving the noise reduction effect, and reduces the repeated noise reduction and excessive noise reduction that may be generated by the two noise reduction modules, reduces power consumption and memory consumption, and improves the problem of excessive loss of image detail information.

[0048] Below Figure 2 Each step in the procedure is described in detail.

[0049] refer to Figure 2 In step S210, the second denoising module obtains the second image and the first reference information outputted after the first denoising module performs denoising on the first image.

[0050] In this exemplary embodiment, the first image, the second image, and the third image described below are image data corresponding to the same frame image in different processing links. For example, the first image may be the RAW data of the current frame image in the video or the RGB data after demosaicing, and the second image is the data of the current frame image output by the first noise reduction module.

[0051] In one implementation, the first noise reduction module may be an ISP noise reduction module, for example, which may perform RAW domain noise reduction on the first image, or may perform primary noise reduction in the RGB domain on the first image. Correspondingly, the second noise reduction module may be a software noise reduction module running in the CPU, which may perform further noise reduction in the RGB domain on the second image.

[0052] The first reference information may be intermediate information obtained when the first noise reduction module performs noise reduction processing on the first image, and the first reference information may provide a reference for the noise reduction processing of the second noise reduction module.

[0053] In one embodiment, the first reference information may include statistical information of the first image, including but not limited to histogram information (Hist), average pixel value (Avg), maximum pixel value (Max), minimum pixel value (Min), pixel variance (Var), pixel gradient (Grad), etc. The statistical information may be used to provide a reference for the noise reduction processing of the first noise reduction module and the noise reduction processing of the second noise reduction module. For example, the first noise reduction module may divide the first image into different image blocks according to the histogram information, and then perform noise reduction processing on different image blocks to different degrees.

[0054] In one embodiment, the first reference information may include difference information between the first image and a reference image of the first image. The reference image may be one or more frames of the first n frames of the first image, such as the previous frame of the first image. The difference information may include any aspect of the difference information between the first image and the reference image. For example, the same indicators (such as Hist, Avg, Max, etc.) are used to perform statistics on the first image and the reference image, and the difference between the statistical indicator values ​​of the two frames of images is calculated, and the difference is added to the first reference information. Alternatively, the reference image may be a frame or multiple frames of the first image, such as the previous frame of the first image. Figure 3 As shown, the first image and the reference image are divided into image blocks according to the same position, that is, the image blocks of the first image correspond to the image blocks of the reference image one by one; for the image blocks in the first image, the difference information between the image blocks and the corresponding image blocks in the reference image is calculated, including the difference of indicators such as Hist, Avg, and Max of the image blocks. The overall difference of the image blocks can be obtained by combining the differences of multiple indicators (for example, the differences of the above multiple indicators can be weighted averaged), for example Figure 3 The overall difference of the image block is 3; based on the same method, the difference information of each image block in the first image can be calculated to obtain Figure 3 The block difference image shown in FIG. 1 is added to the first reference information. Alternatively, reference Figure 3 As shown, the pixel difference information of the first image and the reference image is calculated, such as subtracting the RAW data corresponding to the first image from the RAW data of the reference image, or subtracting the RGB data corresponding to the first image from the RGB data of the reference image (the two frames of RGB data can be extracted from the same module in the ISP, such as from the demosaicing module to ensure the correspondence of the two frames of RGB data), to obtain a pixel difference image (Ghost Map), and add it to the first reference information. It should be understood that Figure 3 The block difference image and the pixel difference image only show the difference of some image blocks or pixels. In practical applications, the difference of each image block or pixel can be calculated.

[0055] The difference information can reflect the degree of motion of the first image relative to the reference image. The first noise reduction module can calculate the difference information before performing noise reduction processing on the first image, and set relevant parameters of the noise reduction processing according to the difference information. In addition, the difference information can also provide an effective reference for the noise reduction processing (especially time domain noise reduction) of the second noise reduction module.

[0056] In one embodiment, the first reference information may include statistical information of the second image, or difference information between the second image and the reference image. The statistical information or difference information of the second image may refer to relevant content of the statistical information or difference information of the first image.

[0057] In one implementation, the first reference information may include configuration information of the first noise reduction module for performing noise reduction processing on the first image, such as which noise reduction methods are used, which types of noise are subjected to noise reduction processing, noise reduction strength information, etc. Based on the first reference information, the second noise reduction module may use a noise reduction method different from that of the first noise reduction module, may perform noise reduction processing on other types of noise, and when the noise reduction strength of the first noise reduction module is relatively large, the second noise reduction module may appropriately reduce the noise reduction strength.

[0058] Continue to refer Figure 2 In step S220, the second denoising module performs denoising on the second image based on the first reference information, and outputs a denoised image corresponding to the first image.

[0059] As can be seen from the above, the first reference information may include statistical information of the first image (or the second image), difference information between the first image (or the second image) and the reference image, configuration information of the first noise reduction module for noise reduction processing, etc. The second noise reduction module may use more appropriate configuration information for noise reduction processing based on the first reference information. Specifically, the second noise reduction module may determine the area-related information and intensity-related information for noise reduction processing based on the above statistical information or difference information; the second noise reduction module may determine to use a noise reduction method not used by the first noise reduction module based on the configuration information of the above first noise reduction module, or perform noise reduction processing on the noise type not processed by the first noise reduction module, etc.

[0060] After the second denoising module performs denoising, it outputs a denoised image corresponding to the first image. The denoised image is the result of the superposition of the denoising processes of the first denoising module and the second denoising module, and can be used as the final output image. Since the first denoising module and the second denoising module are effectively combined in this exemplary embodiment, it is conducive to achieving a high-quality denoising effect. In addition, when performing denoising, the second denoising module can directly call the first reference information, and there is no need to repeat the calculation of the statistical information, difference information, etc., thereby realizing the sharing of intermediate data between the first denoising module and the second denoising module, reducing power consumption and memory consumption.

[0061] In one embodiment, reference Figure 4 As shown, the above-mentioned performing noise reduction processing on the second image based on the first reference information and outputting the noise reduction image corresponding to the first image may include the following steps S410 to S430:

[0062] Step S410, determining spatial domain noise reduction information and temporal domain fusion information based on the first reference information;

[0063] Step S420, performing spatial denoising processing on the second image using the spatial denoising information to obtain a third image;

[0064] Step S430: fuse the third image with the reference image of the second image using the time domain fusion information, and output a denoised image corresponding to the first image.

[0065] Among them, the spatial noise reduction information is used for spatial noise reduction processing, and the temporal fusion information is used for temporal noise reduction processing, that is, the second noise reduction module can adopt a noise reduction method that combines spatial and temporal domains.

[0066] The present disclosure does not limit the specific method of spatial noise reduction processing, such as Gaussian filtering, mean filtering, edge-preserving filtering (specifically, bilateral filtering, guided filtering), etc. In one embodiment, the second noise reduction module uses NLM (Non-Local Mean) filtering to perform spatial noise reduction processing. Spatial noise reduction information may include NLM filtering parameters, such as search window size, neighborhood window size, etc. In general, the higher the required spatial noise reduction intensity, the larger the search window. In step S410, based on the statistical information of the first image (or the second image) in the first reference information, the difference information, or the noise reduction intensity information of the first noise reduction module, the noise reduction intensity required for the second image can be estimated, and even the noise reduction intensity required for different regions (i.e., image blocks) in the second image can be estimated, thereby determining the corresponding search window size, and then determining the appropriate neighborhood window size according to the search window size. In step S420, the search window size and the neighborhood window size are set, and the second image is subjected to NLM filtering to obtain a third image.

[0067] In one embodiment, different noise reduction methods can also be used for different image blocks of the second image, and the spatial noise reduction information includes relevant information on which noise reduction method is used for each image block. Exemplarily, the first reference information may include difference information of each image block in the first image (or the second image). In step S410, according to the difference size of each image block, the motion foreground image block and the background image block are determined in the second image. For example, if the difference of a certain image block in the first image is greater than a predetermined threshold, the corresponding image block in the second image is a motion foreground image block. If the difference of a certain image block in the first image is less than a predetermined threshold, the corresponding image block in the second image is a background image block. For the motion foreground image block, a higher intensity noise reduction method is used, such as NLM filtering, and for the background image block, a normal intensity noise reduction method is used, such as guided filtering. In step S420, each image block in the second image is subjected to spatial noise reduction processing using the corresponding noise reduction method, and a certain smoothing process can also be performed on the boundary between different image blocks to obtain a third image.

[0068] In one embodiment, the second noise reduction module can also obtain inertial sensor data, including acceleration data, gyroscope data, etc., and determine the spatial noise reduction information by combining the inertial sensor data with the first reference information. For example, the acceleration data and gyroscope data collected synchronously when the first image is taken are obtained, and the degree of motion of the electronic device can be estimated based on the acceleration data and the change in the gyroscope data corresponding to the first image and the reference image (such as the change in the gyroscope data corresponding to the last two frames of the image), and the difference information in the first reference information reflects the degree of motion of the first image. Combining the two aspects of the degree of motion can more accurately estimate the degree of motion of different image blocks in the second image, and then determine the appropriate spatial noise reduction information, such as which noise reduction method and corresponding noise reduction parameters are used for each image block.

[0069] Temporal noise reduction processing refers to fusing the reference image of the second image to the third image to eliminate inter-frame noise. Temporal fusion parameters may include the weight of the reference image or the third image used for fusion. Exemplarily, in step S410, based on the statistical information, difference information, or noise reduction strength information of the first image (or the second image) in the first reference information, the degree of motion of the first image (equivalent to the degree of motion of the third image) can be determined, and even the degree of motion of different image blocks or different pixels in the first image (equivalent to the degree of motion of different image blocks or different pixels in the third image) can be determined. The higher the degree of motion, the smaller the weight of the reference image and the larger the weight of the third image. For example, according to the difference information of different image blocks in the first image in the first reference information, the weight α corresponding to the different image blocks is determined, which is the weight of the reference image and is negatively correlated with the difference information. In step S430, fusion can be performed according to the following formula:

[0070] Img denoise =Img Ref ·α+Img3·(1-α) (1)

[0071] Among them, Img denoise Represents the denoised image corresponding to the first image, Img Ref represents the reference image of the above-mentioned second image, and Img3 represents the third image.

[0072] It should be noted that the reference image of the second image may correspond to any one or more frames of the first n frames, and is generally an image processed by the second noise reduction module after time domain noise reduction. The reference image of the first image and the reference image of the second image may correspond to the same frame or different frames. Exemplarily, the reference image of the first image may be an intermediate image when the ISP processes the previous frame, and the reference image of the second image may be an image after the second noise reduction module performs time domain noise reduction on the previous frame.

[0073] In one embodiment, after fusing the third image with the reference image of the second image, the fused image may be post-processed, such as detail recovery, ghost removal, etc., to obtain the final denoised image. In one embodiment, the post-processing may include: fusing the first image or the second image into the fused image to restore the image detail information. For example, the non-flatness of different regions in the second image is detected (non-flatness and flatness are a set of relative concepts. Generally, the higher the signal frequency of a region in the image, the higher the non-flatness of the region and the lower the flatness). For regions whose non-flatness is lower than a predetermined non-flatness threshold, the weight is set to 0. For regions whose non-flatness is not lower than a predetermined non-flatness threshold, the non-flatness is mapped to a weight within [0,1] by numerical mapping, and then the second image is fused into the above-mentioned fused image using the determined weight to obtain a denoised image. The post-processing may be performed by a second denoising module or by other image processing related modules.

[0074] In one implementation, the image noise reduction processing method may further include the following steps:

[0075] The second noise reduction module determines second reference information based on the second image, and outputs the second reference information to the first noise reduction module, so that the first noise reduction module performs noise reduction processing on the first image or other images based on the second reference information.

[0076] The second reference information is used to provide a reference for the noise reduction processing of the first noise reduction module. Exemplarily, the second reference information may include the following two aspects of information:

[0077] ① Configuration information related to noise reduction, such as debugging parameters for the first noise reduction module. For example, the second noise reduction module detects that there is a certain type of noise in the second image, and the first noise reduction module needs to process it (such as the first noise reduction module is more likely to process this type of noise at the hardware level or in the RAW domain), and the second noise reduction module generates configuration information related to the noise type; or the second noise reduction module detects that the detail information of the second image is lost too much, such as the second image has a low gradient or there is obvious blur, and the first noise reduction module needs to reduce the noise reduction strength, and the second noise reduction module generates configuration information related to the noise reduction strength. The first noise reduction module can use the configuration information in the second reference information to adjust relevant parameters or algorithms.

[0078] ② Detection information of the second image (or third image, denoised image). Generally, the second denoising module can be configured with a more complex image detection algorithm, which can detect information that the first denoising module cannot detect, such as image classification information, feature information, image target detection information, semantic segmentation information, etc. For example, the second denoising module performs semantic segmentation on the second image and adds the obtained semantic segmentation image to the second reference information. Based on the semantic segmentation image, the first denoising module can select certain areas in the first image for denoising, or use different denoising methods or denoising parameters for different areas.

[0079] In one implementation, the determining of the second reference information based on the second image may include the following steps:

[0080] The second reference information is determined based on at least one of detail information, motion information, and brightness information of the second image.

[0081] The detail information of the second image may include texture information, high frequency information or quantitative information reflecting the richness of details of the second image; the motion information of the second image may include difference information between the second image and its reference image, image target detection information or semantic segmentation information (image target detection information or semantic segmentation information can reflect which areas in the second image are moving objects); the brightness information of the second image may include statistical information such as brightness and contrast of the second image. The second noise reduction module may determine the second reference information based on one or more of the above information.

[0082] In one embodiment, the second noise reduction module may also obtain inertial sensor data, including acceleration data, gyroscope data, etc., and determine the second reference information in combination with the inertial sensor data and the second image. For example, the second noise reduction module may determine the degree of motion of different image blocks in the second image in combination with the inertial sensor data and the motion information of the second image, and thereby generate configuration information for different image blocks, add the configuration information to the second reference information, and output it to the first noise reduction module, so that the first noise reduction module performs noise reduction processing on the corresponding image blocks in the first image based on the configuration information for different image blocks.

[0083] As can be seen from the above, the second reference information can optimize the noise reduction processing of the first noise reduction module. In one embodiment, the first noise reduction module can re-denoise the first image based on the second reference information. For example, the first noise reduction module and the second noise reduction module can implement iterative processing of the first image. The first noise reduction module first performs noise reduction processing on the first image and outputs the second image and the first reference information; the second noise reduction module can detect the quality of the second image. When the quality meets the predetermined requirements (such as the noise level of the second image is lower than the predetermined noise value, and the gradient of the second image is higher than the predetermined gradient value), the second image is subjected to noise reduction processing based on the first reference information, and the noise-reduced image is output. When the quality does not meet the predetermined requirements, the second reference information is determined and output according to the second image; the first noise reduction module re-denoises the first image according to the second reference information to obtain a new second image and the first reference information; the above process is repeated until the quality of the second image meets the predetermined requirements. In this way, the processing process of the first noise reduction module can be optimized, and the image noise reduction processing effect can be further improved.

[0084] In one implementation, the first noise reduction module may perform noise reduction processing on an image subsequent to the first image (such as a subsequent frame of image) based on the second reference information to optimize noise reduction of subsequent images.

[0085] Figure 5 The schematic flow of the image noise reduction processing method is shown, including:

[0086] Step S501, inputting a first image into a first denoising module, the first denoising module performs denoising processing on the first image based on second reference information, and outputting a second image and the first reference information;

[0087] Step S502, the second noise reduction module determines spatial noise reduction information and temporal fusion information according to the first reference information;

[0088] Step S503: the second denoising module performs spatial denoising processing on the second image using the spatial denoising information to obtain a third image;

[0089] Step S504: the second denoising module fuses the third image with the reference image of the second image using the time domain fusion information to perform time domain denoising on the third image to obtain a fused image;

[0090] Step S505: the second denoising module stores the fused image as a reference image for performing time domain denoising processing on a subsequent frame of image;

[0091] Step S506: the second denoising module determines second reference information according to the second image, and outputs the second reference information to the first denoising module. The second reference information can be used to perform denoising on the first image again, or to perform denoising on a subsequent frame of image.

[0092] Step S507: The second noise reduction module may further perform post-processing on the fused image, and output a noise reduction image corresponding to the first image, and the process ends.

[0093] The exemplary embodiment of the present disclosure also provides another image noise reduction processing method. Figure 6 As shown, the image noise reduction processing method may include the following steps S610 and S620:

[0094] Step S610, the first noise reduction module obtains second reference information output by the second noise reduction module;

[0095] Step S620: the first denoising module performs denoising on the first image based on the second reference information, and outputs the obtained second image to the second denoising module, so that the second denoising module performs denoising on the second image and then outputs a denoised image corresponding to the first image.

[0096] The first noise reduction module and the second noise reduction module run on different processors respectively, for example, the first noise reduction module runs on the ISP and the second noise reduction module runs on the CPU, or the first noise reduction module runs on the CPU and the second noise reduction module runs on the ISP. Alternatively, one of the first noise reduction module and the second noise reduction module is a hardware noise reduction module and the other is a software noise reduction module, for example, the first noise reduction module is a hardware noise reduction module of the ISP, which performs image noise reduction processing at the hardware level, and the second noise reduction module is a software noise reduction module, which performs image noise reduction processing at the software application level, or the first noise reduction module is a software noise reduction module and the second noise reduction module is a hardware noise reduction module of the ISP.

[0097] The second reference information may be noise reduction reference information determined by the second noise reduction module according to any one or more frames of the first n frames of the first image, or noise reduction reference information determined according to the second image output by the first noise reduction module in the previous iteration. The second reference information may include configuration information related to noise reduction, detection information of the second image or the previous frame of the image, etc.

[0098] Based on the above method, by exchanging the second reference information between the first denoising module and the second denoising module, the first denoising module adopts more appropriate configuration information for denoising based on the second reference information, thereby realizing an effective combination of the first denoising module and the second denoising module, which is beneficial to improving the denoising effect, and reducing the repeated denoising, excessive denoising and the like that may be caused by the two denoising modules, reducing power consumption and memory consumption, and improving the problem of excessive loss of image detail information.

[0099] The exemplary embodiment of the present disclosure also provides an image noise reduction processing device. Figure 7 As shown, the image noise reduction processing device 700 may include:

[0100] A first noise reduction module 710 is configured to perform noise reduction processing on the first image and then output a second image and first reference information;

[0101] The second noise reduction module 720 is configured to obtain the second image and the first reference information, perform noise reduction processing on the second image based on the first reference information, and output a noise reduction image corresponding to the first image;

[0102] The first noise reduction module 710 and the second noise reduction module 720 run on different processors respectively; or, one of the first noise reduction module 710 and the second noise reduction module 720 is a hardware noise reduction module, and the other is a software noise reduction module.

[0103] In one implementation, the second noise reduction module 720 is further configured to:

[0104] The second reference information is determined based on the second image, and the second reference information is output to the first noise reduction module 710, so that the first noise reduction module 710 performs noise reduction processing on the first image or other images based on the second reference information.

[0105] In one implementation, the determining of the second reference information based on the second image includes:

[0106] The second reference information is determined based on at least one of detail information, motion information, and brightness information of the second image.

[0107] In one implementation, the above-mentioned performing noise reduction processing on the second image based on the first reference information and outputting the noise reduction image corresponding to the first image includes:

[0108] Determine spatial domain noise reduction information and temporal domain fusion information based on the first reference information;

[0109] Performing spatial denoising processing on the second image using the spatial denoising information to obtain a third image;

[0110] The third image is fused with a reference image of the second image using the time domain fusion information, and a denoised image corresponding to the first image is output.

[0111] In one implementation, the first noise reduction module 710 runs on the ISP, and the second noise reduction module 720 runs on the CPU.

[0112] The exemplary embodiment of the present disclosure also provides another image noise reduction processing device. Figure 8 As shown, the image noise reduction processing device 800 may include:

[0113] The first noise reduction module 810 is configured to obtain the second reference information output by the second noise reduction module 820, perform noise reduction processing on the first image based on the second reference information, and output the obtained second image to the second noise reduction module 820;

[0114] The second noise reduction module 820 is configured to output second reference information, and output a noise reduction image corresponding to the first image after performing noise reduction processing on the second image;

[0115] The first noise reduction module 810 and the second noise reduction module 820 run on different processors respectively; or, one of the first noise reduction module 810 and the second noise reduction module 820 is a hardware noise reduction module, and the other is a software noise reduction module.

[0116] The specific details of each part of the above-mentioned device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0117] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to cause the electronic device to perform the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. In an optional embodiment, the program product can be implemented as a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0118] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0119] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0120] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0121] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0122] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0123] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0124] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image noise reduction processing method, characterized in that: include: The second denoising module acquires a second image and first reference information outputted after the first denoising module performs denoising processing on the first image; The first reference information includes statistical information of the first image, difference information between the first image and a reference image of the first image, or configuration information of the first denoising module for performing denoising processing on the first image; The reference image is one or more frames of images among the first n frames of images of the first image; The second denoising module performs denoising on the second image based on the first reference information, and outputs a denoised image corresponding to the first image; The first noise reduction module and the second noise reduction module are respectively run on different processors; or, one of the first noise reduction module and the second noise reduction module is a hardware noise reduction module, and the other is a software noise reduction module; The performing noise reduction processing on the second image based on the first reference information and outputting a noise-reduced image corresponding to the first image includes: Determine spatial noise reduction information and temporal fusion information based on the first reference information; Performing spatial noise reduction processing on the second image using the spatial noise reduction information to obtain a third image; the first image, the second image, and the third image are image processing corresponding to different processing links of the same frame image; The third image is fused with a reference image of the second image using the time domain fusion information, and a denoised image corresponding to the first image is output.

2. The method according to claim 1, characterized in that The method further comprises: The second denoising module determines second reference information based on the second image, and outputs the second reference information to the first denoising module, so that the first denoising module performs denoising processing on the first image or other images based on the second reference information.

3. The method according to claim 2, characterized in that The determining the second reference information based on the second image comprises: The second reference information is determined based on at least one of detail information, motion information, and brightness information of the second image.

4. The method according to claim 1, characterized in that: The first noise reduction module runs on an image signal processor, and the second noise reduction module runs on a central processing unit.

5. An image noise reduction method, characterized in that: include: The first noise reduction module obtains second reference information output by the second noise reduction module; The first denoising module performs denoising on the first image based on the second reference information, and outputs the obtained second image to the second denoising module, so that the second denoising module performs denoising on the second image and then outputs a denoised image corresponding to the first image; wherein the second denoising module is further configured to: obtain the second image and the first reference information outputted after the first denoising module performs denoising on the first image, and perform denoising on the second image based on the first reference information, and output the denoised image corresponding to the first image; the first reference information includes statistical information of the first image, difference information between the first image and a reference image of the first image, or information of the first denoising module. The configuration information of performing noise reduction processing on the first image by a block; the reference image is one or more frames of images among the first n frames of images of the first image; the noise reduction processing on the second image based on the first reference information, and outputting the noise reduction image corresponding to the first image, is configured as follows: determining spatial noise reduction information and temporal fusion information based on the first reference information; performing spatial noise reduction processing on the second image using the spatial noise reduction information to obtain a third image; the first image, the second image, and the third image are image processing corresponding to different processing links of the same frame image; fusing the third image with the reference image of the second image using the temporal fusion information, and outputting the noise reduction image corresponding to the first image; The first noise reduction module and the second noise reduction module are respectively run on different processors; or, one of the first noise reduction module and the second noise reduction module is a hardware noise reduction module, and the other is a software noise reduction module.

6. An image noise reduction processing device, characterized in that: include: A first noise reduction module is configured to perform noise reduction processing on the first image and then output a second image and first reference information; The first reference information includes statistical information of the first image, difference information between the first image and a reference image of the first image, or configuration information of the first denoising module for performing denoising processing on the first image; The reference image is one or more frames of images among the first n frames of images of the first image; A second noise reduction module is configured to obtain the second image and the first reference information, perform noise reduction processing on the second image based on the first reference information, and output a noise reduction image corresponding to the first image; The first noise reduction module and the second noise reduction module are respectively run on different processors; or, one of the first noise reduction module and the second noise reduction module is a hardware noise reduction module, and the other is a software noise reduction module; The performing noise reduction processing on the second image based on the first reference information and outputting the noise reduction image corresponding to the first image is configured as follows: Determine spatial noise reduction information and temporal fusion information based on the first reference information; Performing spatial noise reduction processing on the second image using the spatial noise reduction information to obtain a third image; the first image, the second image, and the third image are image processing corresponding to different processing links of the same frame image; The third image is fused with a reference image of the second image using the time domain fusion information, and a denoised image corresponding to the first image is output.

7. An image noise reduction processing device, characterized in that: include: A first noise reduction module is configured to obtain second reference information output by the second noise reduction module, perform noise reduction processing on the first image based on the second reference information, and output the obtained second image to the second noise reduction module; The second denoising module is configured to output the second reference information, and output a denoised image corresponding to the first image after performing denoising on the second image; wherein the second denoising module is further configured to: obtain the second image and the first reference information output after the first denoising module performs denoising on the first image, and perform denoising on the second image based on the first reference information, and output the denoised image corresponding to the first image; the first reference information includes statistical information of the first image, difference information between the first image and a reference image of the first image, or a configuration of the first denoising module performing denoising on the first image information; the reference image is one or more frames of the first n frames of the first image; the denoising process is performed on the second image based on the first reference information, and the denoising process is outputted corresponding to the first image, and the configuration is as follows: determining spatial denoising information and temporal fusion information based on the first reference information; performing spatial denoising process on the second image using the spatial denoising information to obtain a third image; the first image, the second image, and the third image are image processing corresponding to different processing links of the same frame image; the third image is fused with the reference image of the second image using the temporal fusion information, and the denoising process is outputted corresponding to the first image; The first noise reduction module and the second noise reduction module are respectively run on different processors; or, one of the first noise reduction module and the second noise reduction module is a hardware noise reduction module, and the other is a software noise reduction module.

8. A computer-readable storage medium, characterized in that: The invention comprises a computer program, which implements the method according to any one of claims 1 to 5 when being executed by a processor.

9. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; The processor is configured to implement the method according to any one of claims 1 to 5 by executing the executable instructions.

Images