Noise reduction method, device, electronic device, medium and computer program product

Abstract

The present application discloses a noise reduction method, device, electronic device, medium and computer program product, belonging to the field of image processing technology. The method comprises: acquiring a first image and a second image, wherein the first image and the second image are obtained by photographing the same scene at the same time, the first image is a color image in the RAW domain, and the second image comprises at least one of the following: a black and white image, a depth of field image, and N is a positive integer; fusing the first image and the second image to obtain a third image; and performing noise reduction on the third image to obtain a fourth image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a noise reduction method, device, electronic equipment, medium and computer program product. Background Art

[0002] With the rapid development of the Internet and technology, Virtual Reality (VR) and Augmented Reality (AR) have gradually become popular technologies in recent years. Among them, wearable devices that combine VR and AR technologies can be called Extended Reality Headsets (XR). XR headsets can display images of simulated real environments on the display of XR headsets through Video See Through (VST) technology.

[0003] At present, in order to improve the clarity of images of simulated real environments, XR headsets usually reduce the noise of images of simulated real environments before transmitting them to displays.

[0004] However, since the images output by the image sensor in the XR headset are usually in YUV format, and after the YUV format images are processed by the image signal processor (ISP) for noise reduction, there is still a lot of noise, and the shape of the noise will change. Therefore, the noise reduction effect of the image is poor. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a noise reduction method, device, electronic device, medium and computer program product, which can improve the noise reduction effect of an image.

[0006] In a first aspect, an embodiment of the present application provides a noise reduction method, the method comprising: acquiring a first image and a second image, the first image and the second image being captured at the same time and of the same scene, the first image being a color image in the original RAW domain, and the second image comprising at least one of the following: a black and white image, a depth of field image; fusing the first image and the second image to obtain a third image; and performing noise reduction on the third image to obtain a fourth image.

[0007] In a second aspect, an embodiment of the present application provides a noise reduction device, which includes: an acquisition module and a processing module; the acquisition module is used to acquire a first image and a second image, the first image and the second image are obtained by shooting the same scene at the same time, the first image is a color image in the original RAW domain, and the second image includes at least one of the following: a black and white image, a depth of field image; the processing module is used to fuse the first image and the second image acquired by the acquisition module to obtain a third image; the processing module is also used to reduce noise on the third image to obtain a fourth image.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, a first image and a second image are obtained, the first image and the second image are obtained by photographing the same scene at the same time, the first image is a color image in the original RAW domain, and the second image includes at least one of the following: a black and white image, a depth of field image, N is a positive integer; the first image and the second image are fused to obtain a third image; and the third image is denoised to obtain a fourth image. Through this scheme, on the one hand, since the image in the RAW domain has not undergone nonlinear image processing, the noise in the image can be modeled more accurately in the RAW domain, thereby achieving a better noise reduction effect. On the other hand, since the number of photons received per unit pixel area in the black and white image is large, and the depth of field image has depth of field information, therefore, after the first image and the second image are fused, noise reduction is performed, and the noise reduction effect of the image can be improved by increasing the number of photons per unit pixel area or performing noise reduction based on depth of field information. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1This is one of the flow charts of a noise reduction method provided in an embodiment of the present application;

[0014] Figure 2 This is the second flowchart of a noise reduction method provided in an embodiment of the present application;

[0015] Figure 3 This is the third flowchart of a noise reduction method provided in an embodiment of the present application;

[0016] Figure 4 This is a fourth flow chart of a noise reduction method provided in an embodiment of the present application;

[0017] Figure 5 This is the fifth flow chart of a noise reduction method provided in an embodiment of the present application;

[0018] Figure 6 This is the sixth flowchart of a noise reduction method provided in an embodiment of the present application;

[0019] Figure 7 is a structural schematic diagram of a noise reduction device provided in an embodiment of the present application;

[0020] Figure 8 This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;

[0021] Fig. 9 This is the second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0023] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0024] The terms "at least one (item)", "at least one of" and the like in the specification of the present application refer to any one, any two or a combination of more than two of the objects contained therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)". The logo in the present application is text, symbol, image, etc. used to indicate information, and logos or other containers can be used as carriers for displaying information, including but not limited to text logos, image logos, symbol logos, etc.

[0025] The noise reduction method, device, electronic device, medium and computer program product provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0026] The noise reduction method, device, electronic device, medium and computer program product provided in the embodiments of the present application can be applied to image noise reduction processing, especially in image processing scenarios in the field of virtual reality technology, such as multi-camera collaborative noise reduction of perspective images in a perspective application scenario using a virtual reality device.

[0027] The noise reduction method, device, electronic device, medium and computer program product provided in the embodiments of the present application, on the one hand, because the image in the RAW domain has not undergone nonlinear image processing, the noise in the image can be modeled more accurately in the RAW domain, thereby achieving a better noise reduction effect. On the other hand, because the number of photons received per unit pixel area in a black and white image is large, and the depth of field image has depth of field information, therefore, after the first image and the second image are fused, noise reduction can be performed, and the noise reduction effect of the image can be improved by increasing the number of photons per unit pixel area or performing noise reduction based on depth of field information.

[0028] The execution subject of the noise reduction method provided in the embodiment of the present application may be a noise reduction device. Exemplarily, the noise reduction device may be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The noise reduction method provided in the embodiment of the present application will be exemplarily described below by taking an electronic device as an example.

[0029] The present application embodiment provides a noise reduction method. Figure 1 A flowchart of a noise reduction method provided by an embodiment of the present application is shown, and the method can be applied to electronic devices. Figure 1 As shown, the noise reduction method provided in the embodiment of the present application may include the following steps 101 to 103.

[0030] Step 101: An electronic device acquires a first image and a second image.

[0031] The first image and the second image may be obtained by photographing the same scene at the same time.

[0032] In the embodiment of the present application, the first image may be a color image in the original (RAW) domain.

[0033] In the embodiment of the present application, the second image may include at least one of the following: a black and white image, a depth of field image.

[0034] In some embodiments of the present application, the above-mentioned electronic device may be a device in a VST for capturing images of a real environment.

[0035] Exemplarily, the above-mentioned electronic device may be an XR device, such as an XR headset.

[0036] In some embodiments of the present application, the electronic device can be used to collect and display images, or can be used to receive images collected by other devices for display. This is not specifically limited in the embodiments of the present application.

[0037] It is understandable that in VST, the electronic device can obtain a real-time augmented reality content video stream by superimposing a virtual image on the captured image of the real environment, and the user can watch the real-time augmented reality content video stream through the display screen of the electronic device. Among them, VST can also be called passthrough, and the user's use of an electronic device (such as an XR headset) for passthrough can be understood as: the user views the surrounding real environment through the display screen of the electronic device.

[0038] In some embodiments of the present application, in order to improve the clarity of the real environment simulated by VST, the electronic device usually performs software noise reduction on the captured image before sending it to the display screen for display. It can be understood that if the image noise reduction effect is poor, the effect of the image displayed on the electronic device display screen will also be poor.

[0039] In some embodiments of the present application, the first image may be an RGB image in the RAW domain.

[0040] It can be understood that the image in the RAW domain may refer to an original image obtained by converting a captured light source signal into a digital signal by a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor.

[0041] In some embodiments of the present application, the first image may be acquired by using an RGB camera.

[0042] Exemplarily, the electronic device may acquire an RGB image in the RAW domain through an RGB camera.

[0043] In some embodiments of the present application, the black-and-white image in the RAW domain may be acquired by a fisheye camera.

[0044] In some embodiments of the present application, the fisheye camera may be a mono fisheye camera. It is understandable that, through mono, the fisheye camera can output black and white images but cannot output color images.

[0045] In some embodiments of the present application, the black-and-white image may be an image in a RAW domain, and the number of the black-and-white images may be N, where N is a positive integer.

[0046] In some embodiments of the present application, the above N may refer to the number of fisheye cameras that capture black-and-white images in the RAW domain.

[0047] Exemplarily, the electronic device may use N fisheye cameras to shoot the same scene to obtain N black-and-white images in the RAW domain.

[0048] In actual implementation, the electronic device may also obtain the black-and-white image in the RAW domain in any other possible manner. This embodiment of the present application does not specifically limit this.

[0049] In some embodiments of the present application, the depth of field image refers to the effect that a certain part of the image captured is clear and the images within a certain range before and after the part are also relatively clear by adjusting the camera aperture, focal length, and distance to the object during the image capture process. That is, the image contains a clear area.

[0050] It is understood that the depth of field (DoF) can refer to the distance range of the object in front and behind that can be measured by the camera or other imager to obtain a clear image. The factors affecting the depth of field mainly include the lens aperture, the lens focal length, and the distance from the focal plane to the object. Through the depth of field, the foreground and background in the image can be distinguished.

[0051] In some embodiments of the present application, the depth of field image may be acquired by a time of flight camera (TOF camera).

[0052] In some embodiments of the present application, the time-of-flight camera may include a time-of-flight sensor, which may be a sensor that uses direct flight time to measure distance. The working principle of the time-of-flight sensor is: based on measuring the time difference between the light pulse emitted and reflected from the object, the distance between the sensor and the object is calculated to obtain a depth of field image.

[0053] In actual implementation, the electronic device can also obtain the depth of field through other devices that can detect the depth of field. This embodiment of the present application is not specifically limited.

[0054] In some embodiments of the present application, when the first image and the second image are captured, the first image and the second image may be captured by a camera of the same device or by cameras of different devices. This embodiment of the present application does not specifically limit this.

[0055] For example, the electronic device is an XR head display. The electronic device may include an RGB camera, N fisheye cameras, and a time of flight camera (TOF camera). The electronic device may obtain N RGB images in RAW domains through the RGB camera; obtain N black and white images in RAW domains through the N fisheye cameras; and obtain a depth of field image through the TOF camera.

[0056] It should be noted that the image in the embodiments of the present application can also be understood as image data, and the two have the same meaning.

[0057] Step 102: The electronic device fuses the first image and the second image to obtain a third image.

[0058] In some embodiments of the present application, before performing image noise reduction processing, the electronic device may first fuse the color image and the black-and-white image in the RAW domain, or the color image and the depth-of-field image in the RAW domain, or the color image, the black-and-white image and the depth-of-field image in the RAW domain. The black-and-white image may be used to increase the brightness in the first image to reduce the noise in the image, or the depth-of-field information may be combined to perform noise reduction to improve the image effect of the noise reduction.

[0059] In some embodiments of the present application, when the second image comprises a black and white image, Figure 1 ,like Figure 2 As shown, the above step 102 can be specifically implemented through the following steps 102a to 102c.

[0060] Step 102a: The electronic device determines a first black-and-white image block from each black-and-white image to obtain N first black-and-white image blocks.

[0061] The first black-and-white image block may be an image block in the black-and-white image where the first black-and-white image block is located, and whose similarity with the first image block in the first image is greater than or equal to a first threshold.

[0062] In some embodiments of the present application, the first image may include multiple image blocks, and the first image block may be any image block among the multiple image blocks.

[0063] It can be understood that each of the N black-and-white images may also include multiple image blocks, and the sizes of the multiple image blocks are the same as the size of the first image block.

[0064] In some embodiments of the present application, for each black-and-white image, the electronic device can determine a first black-and-white image block.

[0065] Exemplarily, taking N=4 as an example, the N black-and-white images may include black-and-white image 1, black-and-white image 2, black-and-white image 3, and black-and-white image 4. The electronic device may determine from black-and-white image 1 a first black-and-white image block 1 whose similarity to the first image block in the first image is greater than or equal to a first threshold; determine from black-and-white image 2 a first black-and-white image block 2 whose similarity to the first image block in the first image is greater than or equal to a first threshold; determine from black-and-white image 3 a first black-and-white image block 3 whose similarity to the first image block in the first image is greater than or equal to a first threshold; and determine from black-and-white image 4 a first black-and-white image block 4 whose similarity to the first image block in the first image is greater than or equal to a first threshold. That is, four first black-and-white image blocks are obtained.

[0066] In some embodiments of the present application, the first threshold may be set by a user, or may be a default threshold of the electronic device, or may be dynamically determined by the electronic device. The present application does not specifically limit this.

[0067] Exemplarily, when the first threshold is dynamically determined by the electronic device, the electronic device may determine the value of the similarity of the image block with the greatest similarity to the first image block in the first image as the first threshold. In other words, the first black-and-white image block may be the image block in the black-and-white image in which it is located that is most similar to the first image block of the first image.

[0068] It should be noted that the first image may also include but is not limited to a second image block, a third image block, a fourth image block... For each image block in the first image, the electronic device can determine a black and white image block whose similarity is greater than or equal to a threshold from each black and white image.

[0069] Step 102b: The electronic device determines a first fusion weight based on the first image block and the N first black-and-white image blocks.

[0070] In some embodiments of the present application, the first fusion weight may be a fusion weight of the first image block and the N first black-and-white image blocks.

[0071] In some embodiments of the present application, Figure 2 ,like Figure 3 As shown, the above step 102b can be specifically implemented through the following step 102b1.

[0072] Step 102b1: The electronic device determines a first fusion weight based on the variance corresponding to the first image block and the variance corresponding to each of the N black-and-white image blocks.

[0073] In some embodiments of the present application, the variance of the above-mentioned image block can be used to measure the uniformity of the gray value distribution of the image.

[0074] It can be understood that a high variance of an image block may indicate that the grayscale value distribution of the image is relatively dispersed, that is, the image contrast is high; a low variance of an image block may indicate that the grayscale value distribution of the image is relatively concentrated, that is, the image contrast is low.

[0075] In some embodiments of the present application, the electronic device may calculate the first fusion weight using the following formula (1).

[0076]

[0077] Wherein, i may represent the index of the image block in the first image, that is, i=0 may represent the first image block in the first image, i=1 may represent the second image block in the first image, and so on; w i can represent the fusion weight corresponding to the i-1th image block in the first image, that is, when i=0, it can represent the first fusion weight corresponding to the first image block; j can represent the variance index, where j=0 can represent the variance of the i-1th image block, j=1 can represent the variance of the first black-and-white image block, j=2 can represent the variance of the first second black-and-white image block, and so on; σ i It can represent variance.

[0078] In this way, the electronic device can flexibly determine the fusion weight of the image block through the variance of the image block to enhance the visual effect of the image, thereby achieving a better noise reduction effect.

[0079] Step 102c: The electronic device fuses the first image block and the N first black-and-white image blocks based on the first fusion weight to obtain a third image.

[0080] In some embodiments of the present application, for each image block in the first image, the electronic device can determine a corresponding fusion weight, so that the electronic device can fuse different image blocks and their corresponding N black-and-white image blocks based on different fusion weights to obtain a third image.

[0081] Exemplarily, taking the first image including four image blocks, which are the first image block, the second image block, the third image block and the fourth image block as an example, the electronic device can fuse the first image block with N first black-and-white image blocks based on the first fusion weight; fuse the second image block with N second black-and-white image blocks based on the second fusion weight; fuse the third image block with N third black-and-white image blocks based on the third fusion weight; and fuse the fourth image block with N fourth black-and-white image blocks based on the fourth fusion weight, to finally obtain the third image.

[0082] In some embodiments of the present application, the electronic device may fuse the first image block and the N first black-and-white image blocks based on the first fusion weight by using the following formula (2).

[0083]

[0084] Among them, R j can represent the image block before fusion, and R can represent the image block after fusion.

[0085] It can be understood that fusing the first image with N black-and-white images is equivalent to the photosensitive element of the camera that took the first image receiving more photons, increasing the signal-to-noise ratio, thereby improving the image quality and making the image after noise reduction better.

[0086] Exemplarily, taking N=4 as an example, fusing the first image with four black-and-white images is equivalent to collecting four times more photons per pixel in the first image, which is equivalent to a 6-dB improvement in signal-to-noise ratio.

[0087] It should be noted that the above image fusion may be a fusion performed on the brightness channel of the first image.

[0088] Exemplarily, taking the first image as an RGB image, the electronic device may perform image fusion processing on the G channel of the first image.

[0089] In some embodiments of the present application, after acquiring a first image and N black-and-white images, the electronic device may first acquire the brightness component and the chromaticity component of the first image, then fuse the brightness component of the first image with the N black-and-white images, and then obtain a third image based on the chromaticity component and the fused brightness component.

[0090] It should be noted that the first image may be divided into a plurality of image blocks, and the electronic device may perform the above steps 102a to 102c on each of the plurality of image blocks to obtain a fused third image.

[0091] In this way, fusing the first image with the N black-and-white images is equivalent to increasing the number of photons received by a unit pixel in the first image, thereby increasing the signal-to-noise ratio of the image and reducing the impact of noise on image quality.

[0092] Step 103: The electronic device performs noise reduction on the third image to obtain a fourth image.

[0093] In some embodiments of the present application, the electronic device can perform noise reduction processing on the obtained third image in the RAW domain, so that the noise in the image can be modeled more accurately in the RAW domain, and after obtaining the result of the noise modeling, the noise intensity can be estimated and applied to the noise reduction algorithm.

[0094] It can be understood that, compared with only denoising the first image, the denoising method provided in the embodiment of the present application can significantly reduce the noise points in the image obtained by VST.

[0095] The noise reduction method provided in the embodiment of the present application, on the one hand, since the image in the RAW domain has not undergone nonlinear image processing, the noise in the image can be modeled more accurately in the RAW domain, thereby achieving a better noise reduction effect. On the other hand, since the number of photons received per unit pixel area in a black and white image is large, and the depth of field image has depth of field information, therefore, after the first image and the second image are fused, noise reduction can be performed, and the noise reduction effect of the image can be improved by increasing the number of photons per unit pixel area or performing noise reduction based on depth of field information.

[0096] In some embodiments of the present application, when the second image includes a depth image, Figure 2 ,like Figure 4 As shown, the above step 102 can be specifically implemented through the following steps 102d and 102e.

[0097] Step 102d: The electronic device obtains depth of field information in the second image.

[0098] In some embodiments of the present application, the above-mentioned depth information may refer to the distance information between each object and the camera in the shooting scene corresponding to the image. It can be understood that this information describes the three-dimensional position of the object, that is, the distance of the object from the observation point (ie, the camera).

[0099] Step 102e: The electronic device adds depth of field information to the first image to obtain a third image.

[0100] In some embodiments of the present application, the adding of depth information to the first image may be understood as: adding depth information to each pixel in the first image based on the depth image to obtain a third image.

[0101] In this way, after the third image containing the depth of field information is obtained by fusion, the electronic device can use different noise reduction weights for different depths of field when performing the subsequent noise reduction step to obtain a better noise reduction effect.

[0102] In some embodiments of the present application, Figure 4 ,like Figure 5 As shown, the above step 103 may specifically include the following steps 103a and 103b.

[0103] Step 103a: The electronic device determines a foreground image and a background image in the third image based on the depth of field information.

[0104] In some embodiments of the present application, the electronic device may segment the third image into a foreground image and a background image according to the depth of field information.

[0105] It can be understood that by dividing the foreground image and the background image according to the depth of field, noise reduction can be performed on areas with different attention levels according to different depths of field, thereby improving the noise performance and performance in the image.

[0106] Step 103b: The electronic device uses the first noise reduction weight to perform noise reduction on the foreground image of the third image, and uses the second noise reduction weight to perform noise reduction on the background image of the third image, so as to obtain a fourth image.

[0107] In some embodiments of the present application, the first noise reduction weight may be greater than the second noise reduction weight.

[0108] In some embodiments of the present application, the electronic device can increase the weight of the foreground image and reduce the weight of the background when reducing noise, and build a model based on different weights, which can effectively reduce the noise of the third image.

[0109] It is understandable that the electronic device can perform noise reduction on the foreground image of the third image to reduce the noise of the foreground image of the third image as much as possible to obtain a fourth image with better noise performance.

[0110] In some embodiments of the present application, the electronic device may use a first noise reduction weight to perform Gaussian filtering noise reduction on the foreground image of the third image, and use a second noise reduction weight to perform Gaussian filtering noise reduction on the background image of the third image to obtain a fourth image.

[0111] It is understandable that for Gaussian filtering, when processing the current pixel, pixels around the current pixel are usually weighted averaged, and the weight of the weighted average depends on the distance between the current pixel and the surrounding pixels. The farther the distance, the smaller the weight.

[0112] It should be noted that the electronic device may also use other possible implementations to perform noise reduction processing on the third image. This embodiment of the present application does not specifically limit this.

[0113] In this way, since the signal-to-noise ratios of different depths of field are different, the electronic device can use different noise reduction weights for pixels of different depths of field, so that the fourth image after noise reduction has better noise performance.

[0114] Each of the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more of them can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0115] The noise reduction method provided in the embodiment of the present application is exemplarily described below with specific examples.

[0116] Assume that the electronic device is an XR headset, which includes an RGB camera, four fisheye cameras, and a time-of-flight camera. Figure 6 As shown, the noise reduction method may include the following steps 201 to 207.

[0117] Step 201: The XR headset collects a first image in the RAW domain through an RGB camera, collects a depth of field image through a time-of-flight camera, and collects four black-and-white images in the RAW domain through four fisheye cameras respectively.

[0118] Step 202: The XR headset determines a first black-and-white image block from each black-and-white image to obtain four first black-and-white image blocks.

[0119] Step 203: The XR headset determines a first fusion weight based on the first image block and the four first black-and-white image blocks.

[0120] Step 204: The XR headset fuses the first image block and the four first black-and-white image blocks based on the first fusion weight to obtain a third image.

[0121] Step 205: The XR head display obtains depth information from the depth image.

[0122] Step 206: The XR headset adds depth of field information to the first image to obtain a third image.

[0123] Step 207: The XR headset performs noise reduction on the third image to obtain a fourth image.

[0124] In this way, on the one hand, since the image in the RAW domain has not undergone nonlinear image processing, the noise in the image can be modeled more accurately in the RAW domain, thereby achieving a better noise reduction effect. On the other hand, since the number of photons received per unit pixel area in a black and white image is large, and the depth of field image has depth of field information, after the first image and the second image are fused, noise reduction can be performed by increasing the number of photons per unit pixel area or performing noise reduction based on depth of field information to improve the noise reduction effect of the image. In this way, the clarity of the user's observation of the external real environment through the XR headset is improved.

[0125] The noise reduction method provided in the embodiment of the present application can be executed by a noise reduction device. In the embodiment of the present application, the noise reduction device provided in the embodiment of the present application is described by taking the noise reduction device executing the noise reduction method as an example.

[0126] Figure 7 FIG. 1 shows a possible structural diagram of a noise reduction device involved in an embodiment of the present application. Figure 7 As shown, the noise reduction device 70 may include: an acquisition module 71 and a processing module 72 .

[0127] Among them, the acquisition module 71 is used to acquire a first image and a second image, the first image and the second image are obtained by shooting the same scene at the same time, the first image is a color image in the original RAW domain, and the second image includes at least one of the following: a black and white image, a depth of field image; the processing module 72 is used to fuse the first image and the second image obtained by the acquisition module 71 to obtain a third image; and is used to reduce noise on the third image to obtain a fourth image.

[0128] In a possible implementation, the second image includes a black-and-white image, the black-and-white image is an image in the RAW domain, and the number of the black-and-white images is N, where N is a positive integer;

[0129] The processing module 72 is specifically used for:

[0130] Determine a first black-and-white image block from each black-and-white image to obtain N first black-and-white image blocks, where the first black-and-white image block is: an image block in the black-and-white image where the first black-and-white image block is located and whose similarity with the first image block in the first image is greater than or equal to a first threshold;

[0131] Determining a first fusion weight based on the first image block and the N first black-and-white image blocks;

[0132] Based on the first fusion weight, the first image block and the N first black-and-white image blocks are fused to obtain a third image.

[0133] In a possible implementation, the processing module 72 is specifically configured to determine the first fusion weight based on the variance corresponding to the first image block and the variance corresponding to each of the N black-and-white image blocks.

[0134] In a possible implementation, the second image includes a depth-of-field image;

[0135] The processing module 72 is specifically used for:

[0136] Acquiring depth of field information in the second image;

[0137] Depth of field information is added to the first image to obtain a third image.

[0138] In a possible implementation, the processing module 72 is specifically configured to:

[0139] Determining a foreground image and a background image in a third image based on the depth of field information;

[0140] The first denoising weight is used to reduce the noise of the foreground image of the third image, and the second denoising weight is used to reduce the noise of the background image of the third image, so as to obtain a fourth image.

[0141] The embodiment of the present application provides a noise reduction device. On the one hand, since the image in the RAW domain has not undergone nonlinear image processing, the noise in the image can be modeled more accurately in the RAW domain, thereby achieving a better noise reduction effect. On the other hand, since the number of photons received per unit pixel area in a black and white image is large, and the depth of field image has depth of field information, therefore, after the first image and the second image are fused, noise reduction can be performed, and the noise reduction effect of the image can be improved by increasing the number of photons per unit pixel area or performing noise reduction based on depth of field information.

[0142] The noise reduction device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0143] The noise reduction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0144] The noise reduction device provided in the embodiment of the present application can implement various processes implemented in the above-mentioned noise reduction method embodiment and achieve the same technical effect. To avoid repetition, it will not be described here.

[0145] Alternatively, if Figure 8 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores a program or instruction that can be executed on the processor 801, and when the program or instruction is executed by the processor 801, the various steps of the above-mentioned noise reduction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0146] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0147] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0148] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0149] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0150] Among them, the processor 1010 is used to obtain a first image and a second image, the first image and the second image are obtained by shooting the same scene at the same time, the first image is a color image in the original RAW domain, and the second image includes at least one of the following: a black and white image, a depth of field image; and is used to fuse the first image and the second image obtained by the processor 1010 to obtain a third image; and is also used to reduce noise on the third image to obtain a fourth image.

[0151] In a possible implementation, the second image includes a black-and-white image, the black-and-white image is an image in the RAW domain, and the number of the black-and-white images is N, where N is a positive integer;

[0152] The processor 1010 is specifically configured to:

[0153] Determine a first black-and-white image block from each black-and-white image to obtain N first black-and-white image blocks, where the first black-and-white image block is: an image block in the black-and-white image where the first black-and-white image block is located and whose similarity with the first image block in the first image is greater than or equal to a first threshold;

[0154] Determining a first fusion weight based on the first image block and the N first black-and-white image blocks;

[0155] Based on the first fusion weight, the first image block and the N first black-and-white image blocks are fused to obtain a third image.

[0156] In a possible implementation, the processor 1010 is specifically configured to determine a first fusion weight based on a variance corresponding to the first image block and a variance corresponding to each of the N black-and-white image blocks.

[0157] In a possible implementation, the second image includes a depth-of-field image;

[0158] The processor 1010 is specifically configured to:

[0159] Acquiring depth of field information in the second image;

[0160] Depth of field information is added to the first image to obtain a third image.

[0161] In a possible implementation manner, the processor 1010 is specifically configured to:

[0162] Determining a foreground image and a background image in a third image based on the depth of field information;

[0163] The first denoising weight is used to reduce the noise of the foreground image of the third image, and the second denoising weight is used to reduce the noise of the background image of the third image, so as to obtain a fourth image.

[0164] The embodiment of the present application provides an electronic device. On the one hand, since the image in the RAW domain has not undergone nonlinear image processing, the noise in the image can be modeled more accurately in the RAW domain, thereby achieving a better noise reduction effect. On the other hand, since the number of photons received per unit pixel area in a black and white image is large, and the depth of field image has depth of field information, therefore, after the first image and the second image are fused, noise reduction can be performed, and the noise reduction effect of the image can be improved by increasing the number of photons per unit pixel area or performing noise reduction based on depth of field information.

[0165] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0166] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0167] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0168] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned noise reduction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0169] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0170] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned noise reduction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0171] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0172] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned noise reduction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0173] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0174] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0175] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A noise reduction method, characterized in that: The method comprises: Acquire a first image and a second image, wherein the first image and the second image are obtained by photographing the same scene at the same time, the first image is a color image in the original RAW domain, and the second image includes at least one of the following: a black and white image and a depth of field image; fusing the first image and the second image to obtain a third image; De-noising the third image to obtain a fourth image.

2. The method according to claim 1, characterized in that The second image includes the black-and-white image, the black-and-white image is an image in the RAW domain, and the number of the black-and-white images is N, where N is a positive integer; The fusing the first image and the second image to obtain a third image includes: Determine a first black-and-white image block from each black-and-white image to obtain N first black-and-white image blocks, wherein the first black-and-white image block is an image block in the black-and-white image where the first black-and-white image block is located and whose similarity with the first image block in the first image is greater than or equal to a first threshold value; Determining a first fusion weight based on the first image block and the N first black-and-white image blocks; Based on the first fusion weight, the first image block and the N first black-and-white image blocks are fused to obtain the third image.

3. The method according to claim 2, characterized in that The determining a first fusion weight based on the first image block and the N first black-and-white image blocks includes: The first fusion weight is determined based on the variance corresponding to the first image block and the variance corresponding to each black-and-white image block in the N black-and-white image blocks.

4. The method according to claim 1, characterized in that The second image comprises a depth image; The fusing the first image and the second image to obtain a third image includes: Acquiring depth of field information in the second image; The depth of field information is added to the first image to obtain the third image.

5. The method according to claim 4, characterized in that The step of reducing noise on the third image to obtain a fourth image includes: Based on the depth of field information, determining a foreground image and a background image in the third image; The foreground image of the third image is denoised using the first denoising weight, and the background image of the third image is denoised using the second denoising weight, to obtain the fourth image.

6. A noise reduction device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is used to acquire a first image and a second image, wherein the first image and the second image are obtained by photographing the same scene at the same time, the first image is a color image in the original RAW domain, and the second image includes at least one of the following: a black and white image and a depth of field image; The processing module is used to fuse the first image and the second image acquired by the acquisition module to obtain a third image; The processing module is further used to perform noise reduction on the third image to obtain a fourth image.

7. The device according to claim 6, characterized in that The second image includes the black-and-white image, the black-and-white image is an image in the RAW domain, and the number of the black-and-white images is N, where N is a positive integer; The processing module is specifically used for: Determine a first black-and-white image block from each black-and-white image to obtain N first black-and-white image blocks, wherein the first black-and-white image block is an image block in the black-and-white image where the first black-and-white image block is located and whose similarity with the first image block in the first image is greater than or equal to a first threshold value; Determining a first fusion weight based on the first image block and the N first black-and-white image blocks; Based on the first fusion weight, the first image block and the N first black-and-white image blocks are fused to obtain the third image.

8. The device according to claim 7, characterized in that The processing module is specifically configured to determine the first fusion weight based on the variance corresponding to the first image block and the variance corresponding to each of the N black-and-white image blocks.

9. The device according to claim 6, characterized in that The second image comprises a depth image; The processing module is specifically used for: Acquiring depth of field information in the second image; The depth of field information is added to the first image to obtain the third image.

10. The device according to claim 9, characterized in that The processing module is specifically used for: Based on the depth of field information, determining a foreground image and a background image in the third image; The foreground image of the third image is denoised using the first denoising weight, and the background image of the third image is denoised using the second denoising weight, to obtain the fourth image.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the noise reduction method according to any one of claims 1 to 5 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the noise reduction method according to any one of claims 1 to 5 are implemented.

13. A computer program product, wherein the computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the steps of the noise reduction method according to any one of claims 1 to 5.

Images