Method and device for removing noise

Abstract

The embodiment of the present application provides a method for removing noise, which is applied to an electronic device, wherein the electronic device has an ultra-wide-angle camera, and the ultra-wide-angle camera is used to collect image frames, and the image frames include a first image frame: obtaining a gain map of the first image frame, the gain map includes the noise reduction intensity of the pixel points of the first image frame, and the distance between the pixel points and the center point of the first image frame is positively correlated with the noise reduction intensity. In the process of splicing the image frames into an ultra-wide-angle image, the noise of the image frame is removed based on the gain map and the depth map of the first image frame. Because the noise reduction intensity of the pixel points farther from the center point is greater, the noise in the edge area of ​​the image can be fully eliminated, and because the noise reduction intensity of the pixel points closer to the center point is smaller, more detailed information in the center area of ​​the image can be retained. In summary, the method for removing noise can enable the electronic device to have a better denoising function, and is particularly suitable for noise removal of ultra-wide-angle images.

Description

Technical Field

[0001] The present application relates to the field of electronic information technology, and in particular to a method and device for removing noise. Background Art

[0002] Image capture is a common function of electronic devices. Due to the limitations of shooting technology, the captured images usually contain noise. Therefore, it is necessary to remove noise from the captured images.

[0003] The noise removal capabilities currently used in electronic devices leave room for improvement. Summary of the invention

[0004] The present application provides a method and device for removing noise, aiming to solve the problem of how to improve the noise removal function of electronic equipment.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] The first aspect of the present application provides a method for removing noise, which is applied to an electronic device, wherein the electronic device has an ultra-wide-angle camera, the ultra-wide-angle camera is used to collect image frames, the image frames include a first image frame, and the method for removing noise includes: obtaining a gain map of the first image frame, the gain map includes the noise reduction intensity of the pixel points of the first image frame, the distance between the pixel points and the center point of the first image frame is positively correlated with the noise reduction intensity, and in the process of splicing the image frames into an ultra-wide-angle image, based on the gain map and the depth map of the first image frame, removing the noise of the image frame, wherein the noise reduction intensity of the edge intensity in the ultra-wide-angle image obtained by the image frame after noise removal is greater than the noise reduction intensity of the center area. Because the noise reduction intensity of the pixel points farther from the center point is greater, the noise in the edge area of ​​the image can be fully removed, and because the noise reduction intensity of the pixel points closer to the center point is smaller, more detailed information in the center area of ​​the image can be retained. In summary, the method for removing noise can enable the electronic device to have a better denoising function, and is particularly suitable for noise removal of ultra-wide-angle images.

[0007] In some implementations, the method of obtaining the gain mapping of the first image frame includes: obtaining a basic noise reduction strength, adjusting the basic noise reduction strength according to a rule that the distance between the pixel point and the center point of the first image frame is positively correlated with the noise reduction strength, and obtaining the noise reduction strength of the pixel point. On the basis of the basic noise reduction strength, according to the rule that the distance to the center point is positively correlated, the noise reduction strength of each pixel point is obtained, which has higher convenience and feasibility.

[0008] In some implementations, the method for obtaining the basic noise reduction strength includes: determining a first noise reduction strength based on the noise strength corresponding to the sensitivity of the ultra-wide-angle camera, adjusting the first noise reduction strength based on the light intensity value corresponding to the scene information to obtain the basic noise reduction strength, and the scene information indicates the shooting mode. The first noise reduction strength determined based on the sensitivity is more consistent with the noise strength of the image captured by the ultra-wide-angle camera, and the basic noise reduction strength obtained by adjusting the first noise reduction strength based on the scene information is more consistent with the noise caused by the lighting environment of the shooting environment and the demand for image clarity.

[0009] In some implementations, the basic noise reduction intensity is adjusted according to the rule that the distance between the pixel point and the center point of the first image frame is positively correlated with the noise reduction intensity, and the method for obtaining the noise reduction intensity of the pixel point includes: based on the distance between the pixel point and the center point, the first image frame is divided into multiple areas, the multiple areas include a central area and an edge area, the central area is an area composed of pixels whose distance from the center point is not greater than a first value, the edge area is an area composed of pixels whose distance from the center point is not less than a second value, the first value is not greater than the second value, the noise reduction intensity of the central area is configured as the basic noise reduction intensity, the noise reduction intensity of the edge area is configured as the maximum noise reduction intensity, and the maximum noise reduction intensity is greater than the basic noise reduction intensity. Dividing the ultra-wide-angle image into areas and configuring the noise reduction intensity according to the area has a smaller amount of calculation than configuring the noise reduction intensity pixel by pixel, which is conducive to saving computing resources.

[0010] In some implementations, the multiple regions also include an intermediate region, the intermediate region is an region composed of pixels whose distance from the center point is within a range from a first value to a second value, and the basic noise reduction intensity is adjusted according to the rule that the distance between the pixel point and the center point of the first image frame is positively correlated with the noise reduction intensity to obtain the noise reduction intensity of the pixel point, and further includes: based on the distance between the edge region and the center region, obtaining the change trend between the basic noise reduction intensity and the maximum noise reduction intensity, based on the distance between the pixel point and the center point in the intermediate region and the change trend, adjusting the basic noise reduction intensity to obtain the noise reduction intensity of the pixel point. Configuring the noise reduction intensity pixel by pixel in the intermediate region is conducive to balancing the saving of computing resources and more sophisticated noise removal.

[0011] In some implementations, based on the gain map and the depth map of the first image frame, the method of removing noise from the image frame includes: fusing the gain map with the depth map of the first image frame to obtain a fused image, and removing noise from the image frame based on the fused image. Using the depth map as a basis for denoising is conducive to further improving the accuracy of the noise reduction strength, thereby removing noise more finely.

[0012] In some implementations, the method of fusing the gain map with the depth map of the first image frame to obtain the fused image includes: performing weighted addition of the gain map and the depth map of the first image frame to obtain the fused image. The weighted addition method is easier to implement.

[0013] In some implementations, based on the fused image, the method of removing noise from the image frame includes: inputting the fused image and the image frame into a model to obtain an ultra-wide-angle image output by the model, wherein the model includes a noise reduction model, and the noise reduction model is used to remove noise from the image frame based on the fused model. Noise removal is achieved during the stitching process, which has higher image output efficiency, and removing noise from the image frame can enable the ultra-wide-angle image to obtain a better denoising effect.

[0014] The second aspect of the present application provides an electronic device, comprising: one or more processors, a memory, a display screen and a camera, wherein the memory, the display screen and the camera are coupled to the one or more processors, the memory is used to store computer program code, and the computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the noise removal method provided in the first aspect of the present application.

[0015] The third aspect of the present application provides a computer-readable storage medium having a program stored thereon, which implements the noise removal method provided in the first aspect of the present application when a computer device runs an application program. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is an example of different noise intensities in different areas of an ultra-wide-angle image;

[0017] Figure 2 A flowchart of a method for removing noise disclosed in an embodiment of the present application;

[0018] Figure 3 This is an example diagram of the relationship between iso and Gaussian noise intensity;

[0019] Figure 4 An example diagram of configuring different noise reduction intensities for different areas disclosed in the embodiments of the present application;

[0020] Figure 5 An example diagram of removing noise based on noise reduction strengths in different regions disclosed in an embodiment of the present application;

[0021] Figure 6 It is an example diagram of denoising effect of a conventional method and a method for removing noise disclosed in an embodiment of the present application;

[0022] Figure 7 An example diagram of the structure of an electronic device;

[0023] Figure 8 An example diagram of a software framework running in an electronic device. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0025] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0026] The multiple involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0027] In order to improve the image capture function of electronic devices, electronic devices are usually equipped with multiple types of cameras. For example, in addition to the main camera (such as a camera with a focal length ranging from 40 mm to 60 mm), a mobile phone is also equipped with an ultra-wide-angle camera (the smaller the focal length, the larger the viewing angle. An example of an ultra-wide-angle camera is a camera with a focal length of less than 40 mm). The ultra-wide-angle camera has a larger viewing angle and depth of field, so it can capture targets within a larger viewing angle range.

[0028] During the research, the inventors found that the farther the sensor is from the center point (sensor) of the ultra-wide-angle camera, the weaker the light intake and resolution (which can be understood as the ability to express details). The weakening of light intake and resolution will lead to a higher intensity of noise in the image.

[0029] by Figure 1 Taking the ultra-wide-angle image A taken by the ultra-wide-angle camera as an example, the enlarged image of the central area of ​​the ultra-wide-angle image A is A1, and the enlarged image of the lower right corner area is A2. Because the sensor that takes the lower right corner is far away from the center point (sensor) of the wide-angle camera, the amount of light entering and the resolution are weaker than the center point (sensor). Therefore, the noise intensity of the lower right corner area in the ultra-wide-angle image A is greater than the noise intensity of the center area. Figure 1 In the figure, white dots represent the noise intensity. The number of white dots in A2 is greater than that in A1, which means that the noise intensity in A2 is greater than that in A1.

[0030] The inventors also found during the research that, in view of the characteristics of noise in images taken by ultra-wide-angle cameras, existing noise reduction algorithms are not suitable for ultra-wide-angle images for the following reasons:

[0031] On the one hand, in order to better remove the noise in the edge area of ​​the ultra-wide-angle image (such as A2), it is necessary to remove the noise with a greater intensity, but a greater noise reduction intensity will cause more details in the central area of ​​the ultra-wide-angle image (such as A1) to be removed, that is, the noise reduction intensity suitable for the edge area will cause a greater loss of details in the central area. On the other hand, in order to retain more details in the central area, the noise reduction intensity needs to be reduced, but it will still cause strong noise in the edge area.

[0032] In order to solve the above problems, an embodiment of the present application provides a method for removing noise, which uses different noise reduction intensities for the central area and edge area of ​​the ultra-wide-angle image to remove noise. The farther the area is from the center of the image, the greater the noise reduction intensity, thereby better removing the noise in the edge area while maximizing the detail information of the central area.

[0033] The noise removal method provided in the embodiments of the present application is applied to, but not limited to, the scenario of noise removal of ultra-wide-angle images. The ultra-wide-angle image may be an image obtained in the photo mode of the ultra-wide-angle camera of the electronic device, or may be any frame of image in the video obtained in the video recording mode of the ultra-wide-angle camera. In the scenario of taking photos or recording videos using the ultra-wide-angle camera of an electronic device, before obtaining an image or video to be displayed to a user (such as stored in a gallery of the electronic device), the noise removal method provided in the embodiments of the present application is performed on the image or video taken by the ultra-wide-angle camera to obtain an image or video to be displayed to the user.

[0034] Figure 2 The process of the noise removal method disclosed in the embodiment of the present application includes the following steps:

[0035] S01. Acquire a reference frame of an ultra-wide-angle image.

[0036] It is understandable that the ultra-wide-angle image is obtained by stitching multiple frames of images. One frame of image selected as a reference in the multiple frames of images for stitching is called a reference frame. The reference frame of the ultra-wide-angle image is the reference frame in the multiple frames of images for stitching the ultra-wide-angle image.

[0037] After acquiring multiple frames of images, the electronic device selects a reference frame based on a pre-configured algorithm. The electronic device may store information of the reference frame, and when executing S01, reads the reference frame through the stored information of the reference frame. The method of selecting the reference frame is not described in detail here.

[0038] S02. Obtain a depth map of a reference frame.

[0039] The depth map can be understood as an image composed of the depth information of pixels in the reference frame.

[0040] In some implementations, a depth estimation network is used to perform depth estimation on a reference frame to obtain depth information of each pixel, thereby obtaining a depth map of the reference frame.

[0041] S03. Obtain a gain map of a reference frame.

[0042] Gainmap can be understood as the noise reduction strength of each pixel. The principle of gainmap acquisition is described in S031-S032:

[0043] S031. Obtain basic noise reduction strength based on the detected shooting scene information.

[0044] The shooting scene can be understood as the shooting mode of the camera, and examples of the shooting mode include but are not limited to: night scene and macro, etc. It can be understood that the shooting scene described in this embodiment is a shooting mode under the premise of using an ultra-wide-angle camera.

[0045] In some implementations, after shooting is initiated (eg, a user presses a shooting button), a shooting scene is detected to obtain a shooting scene used for shooting an ultra-wide-angle image.

[0046] In this embodiment, the intensity used to remove the noise in the central area of ​​the ultra-wide-angle image is called the basic noise reduction intensity.

[0047] It is understandable that, because the noise of images obtained in different shooting scenes is different, obtaining the basic noise reduction strength based on the information of the shooting scene is conducive to more targeted noise removal. That is to say, for images taken in a certain shooting scene, it is possible to fully remove the noise and retain more details of the image.

[0048] In some implementations, the specific implementation process of S03 includes the following steps:

[0049] 1. Determine a first noise reduction intensity based on the noise intensity corresponding to the sensitivity (ISO) of the ultra-wide-angle camera.

[0050] The relationship between iso and Gaussian noise intensity is as follows Figure 3 As shown, it can be understood that Gaussian noise is used as an example here, and other noises can also be replaced. The corresponding relationship between other noises and ISO is not repeated here.

[0051] In some implementations, the noise intensity corresponding to the ISO of the ultra-wide-angle camera used to shoot the ultra-wide-angle image is used as the first noise reduction intensity. In other implementations, an adjustment is made based on the noise intensity corresponding to the ISO of the ultra-wide-angle camera used to shoot the ultra-wide-angle image to obtain the first noise reduction intensity.

[0052] 2. Based on the illumination intensity value corresponding to the scene information, the first noise reduction intensity is adjusted to obtain a basic noise reduction intensity.

[0053] The scene information indicates the shooting scene. The light intensity value of the shooting scene represents the lighting conditions under the shooting scene. The lighting conditions are associated with the noise in the captured image. Therefore, adjusting the first noise reduction intensity based on the light intensity value corresponding to the scene information is conducive to obtaining a noise reduction intensity that better matches the shooting scene.

[0054] In some implementations, a correspondence between a shooting scene, a light intensity value, and an adjustment coefficient is configured in an electronic device, the adjustment coefficient is obtained by querying the correspondence, and a basic noise reduction intensity is obtained based on the adjustment coefficient and the first noise reduction intensity.

[0055] Because a larger light intensity value indicates stronger light, the tolerance to noise is lower. Therefore, in some implementations, a larger light intensity value indicates a larger basic noise reduction intensity.

[0056] It can be understood that, because the camera's ISO and the light intensity values ​​corresponding to each shooting mode can be known, the basic noise reduction intensity can be written as a configuration parameter in a configuration file, and the configuration file can be stored in an electronic device. When the basic noise reduction intensity needs to be called, the basic noise reduction intensity can be read from the configuration file.

[0057] S032. Determine the noise reduction intensity of each pixel point based on the distance between the pixel point and the center point in the reference frame and the basic noise reduction intensity, and obtain a gain map of the reference frame.

[0058] For ease of explanation, the first pixel point represents any pixel point in the reference frame.

[0059] The larger the distance value between the first pixel point and the center point of the reference frame, the farther the first pixel point is from the center point. In this embodiment, the noise reduction intensity configured for the first pixel point is greater, so as to fully remove the noise of the first pixel point.

[0060] The smaller the distance value between the first pixel point and the center point of the reference frame, the closer the first pixel point is to the center point. In this embodiment, the noise reduction intensity configured for the first pixel point is smaller, so that the noise of the first pixel point can be fully removed while retaining more details.

[0061] In this embodiment, the center point can be understood as a pixel point located at the center of the image, and can also be understood as any pre-specified pixel point.

[0062] To facilitate implementation and reduce the amount of calculation, in some implementations, the reference frame is divided into a central area, an edge area, and an intermediate area between the central area and the edge area.

[0063] The central area is an area composed of pixel points whose distance from the central point is not greater than the first value. Figure 4 For example, the central area of ​​the reference frame B is B1. It can be understood that the center point is located in the central area and is the center point of the central area. The pixels in the central area are configured with the same noise reduction intensity, and the noise reduction intensity of the central area is the above-mentioned basic noise reduction intensity. Figure 4 The black color in the center indicates the noise reduction strength in the central area.

[0064] The edge area is an area composed of pixel points whose distance from the center point is not less than the second value. Figure 4 For example, the edge area of ​​the reference frame B is B2. The pixels in the edge area are configured with the same noise reduction strength. It can be understood that the noise reduction strength of the pixels in the edge area B2 is greater than the noise reduction strength of the pixels in the center area B1. Figure 4 The white color indicates the noise reduction strength of the edge area. The noise reduction strength of the edge area can be referred to as the maximum noise reduction strength. The maximum noise reduction strength has no upper limit in theory. In combination with practical applications, in this embodiment, it is set to 70%-90% of the basic noise reduction strength.

[0065] The middle area is an area composed of pixels whose distance from the center point is within the range of (first value, second value). Unlike the center area and the edge area, the noise reduction strength of the pixels in the middle area is different, that is, the noise reduction strength of the pixels in the middle area is configured according to the rule that the farther the distance from the center point, the larger the value. It can be understood that the noise reduction strength of the pixels in the middle area is greater than the basic noise reduction strength of the pixels in the center area, and less than the maximum noise reduction strength of the pixels in the edge area.

[0066] by Figure 4 For example, the noise reduction strength of the pixel points in the middle area B3 is between black and white, and the closer to the center point, the closer to black, indicating a smaller noise reduction strength, and the farther from the center point, the closer to white, indicating a larger noise reduction strength.

[0067] In some implementations, based on the distance between the edge area and the center area, the variation trend between the basic noise reduction intensity and the maximum noise reduction intensity is obtained, and then based on the distance between the pixel point in the middle area and the center point and the variation trend, the basic noise reduction intensity is adjusted to obtain the noise reduction intensity of the pixel point. The smaller the distance between the pixel point in the middle area and the center point, the smaller the noise reduction intensity.

[0068] In some implementations, the maximum noise reduction strength, the radius of the central area (i.e., the first value), and the basis for determining the middle area and the edge area (i.e., the second value) can all be written into a configuration file stored in the electronic device as configuration parameters. The configuration file is read during the execution of this step.

[0069] After each pixel is configured with a noise reduction strength, the gainmap of the reference frame is obtained.

[0070] It is understandable that, because the size of the ultra-wide-angle image taken by the electronic device after leaving the factory, the iOS of the ultra-wide-angle camera and the light intensity of each shooting scene are fixed, so in combination with S031 and S032, it can be known that in some implementations, the gainmap of the reference frame is stored in the electronic device in the form of a configuration file as a template. In S03, the gainmap of the reference frame is obtained by reading, so that the noise reduction process can obtain better real-time performance. In other implementations, the gainmap of the reference frame is calculated based on S031 and S032.

[0071] S04. Fusing the depth map of the reference frame with the gain map of the reference frame to obtain a fused image.

[0072] In some implementations, the depth map and the gain map are weighted added to obtain a fused image. Figure 5 For example, the depth map and gainmap are weightedly added to obtain the fused image.

[0073] The depth map represents the distance of each point on the image from the camera. The farther the point is from the camera, the weaker the corresponding light input, so the noise will be greater. Therefore, the subsequent noise removal process based on the depth map is conducive to further obtaining the noise reduction strength of the pixel more accurately, that is, making the noise reduction strength more closely match the actual noise strength of the pixel.

[0074] S05. Perform noise reduction and stitching processing on the image frames based on the fused image to obtain an ultra-wide-angle image.

[0075] like Figure 5 As shown, in some implementations, the fused image and the image frame are used as inputs of the trained model to obtain an ultra-wide-angle image output by the model. The model includes a denoising model and a splicing model. The denoising model is used to remove noise in the image frame based on the fusion model, and the splicing model is used to splice the denoised image frame to obtain the ultra-wide-angle image.

[0076] Figure 2 In the process shown, different noise reduction strengths are configured for pixels in different areas, that is, a higher noise reduction strength is configured for pixels in areas with stronger noise, and a lower noise reduction strength is configured for pixels in areas with weaker noise. This can include as much detail information of the ultra-wide-angle image as possible while removing noise.

[0077] by Figure 6 For example, when using the traditional solution for noise removal, if the details of the central area A1 are to be retained as much as possible, a larger noise reduction intensity cannot be used. Therefore, the noise in the A2 area after being processed by the traditional solution is still relatively large.

[0078] If the edge area A2 is guaranteed to have a better noise removal effect, the details of the central area A1 will be blurred, such as Figure 6 The result of the traditional scheme after processing A1 is shown, and the details are more blurred than before the denoising process.

[0079] And adopt Figure 2 After the noise removal method described above, Figure 6 As shown, the details of the central area A1 are better preserved, and the noise of the edge area A2 is better removed (that is, compared with the processing result of the traditional solution, there are fewer noise points in the processing result of the present invention).

[0080] The noise removal method described in the embodiments of the present application is applied to electronic devices, including but not limited to mobile phones, tablet computers, desktops, laptops, notebook computers, ultra-mobile personal computers (UMPC), handheld computers, netbooks, personal digital assistants (PDA), wearable electronic devices, smart watches and other electronic devices with cameras.

[0081] Figure 7 The present invention provides an example of the composition of an electronic device. Taking a mobile phone as an example, the electronic device may include a processor 110, an internal memory 120, a display screen 130, a camera 140, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, and an audio module 170.

[0082] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0083] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0084] The internal memory 120 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 120. The internal memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 120, and / or the instructions stored in the memory provided in the processor.

[0085] The electronic device implements the display function through a GPU, a display screen 130, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 130 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0086] The display screen 130 is used to display images, videos, etc. The display screen 130 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 130, where N is a positive integer greater than 1.

[0087] The electronic device can realize the shooting function through ISP, camera 140, video codec, GPU, display screen 194 and application processor.

[0088] The ISP is used to process the data fed back by the camera 140. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 140.

[0089] The camera 140 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 140, where N is a positive integer greater than 1.

[0090] The camera 140 includes at least a main camera and an ultra-wide-angle camera.

[0091] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency point, a digital signal processor is used to perform Fourier transform on the frequency point energy.

[0092] Video codecs are used to compress or decompress digital videos. Electronic devices can support one or more video codecs. In this way, electronic devices can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0093] The internal memory 120 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 120. The internal memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 120, and / or the instructions stored in the memory provided in the processor.

[0094] The electronic device can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the microphone 170B and the application processor.

[0095] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0096] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device can listen to music or listen to a hands-free call through the speaker 170A.

[0097] In some embodiments, the speaker 170A can play the video information with special effects mentioned in the embodiments of the present application.

[0098] Microphone 170B, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by approaching the microphone 170B with his mouth to input the sound signal into the microphone 170B.

[0099] In some embodiments, the microphone 170B can collect the sound of the environment in which the electronic device is located when the camera is shooting video information with special effects.

[0100] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0101] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0102] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic devices. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0103] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to electronic devices. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0104] exist Figure 7 An operating system runs on the above components, such as an iOS operating system, an Android operating system, a Windows operating system, etc. Application programs can be installed and run on the operating system.

[0105] Figure 8 It is a software structure block diagram of the electronic device according to an embodiment of the present application.

[0106] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0107] The application layer can include a series of application packages. Figure 8 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, image processing, etc.

[0108] In some embodiments, the camera is used to take photos or videos. After the camera takes the photos or videos, the electronic device can jump to execute the image processing application, and the image processing application processes the image. The image processing includes but is not limited to denoising the image using the noise removal method disclosed in the embodiment of the present application and stitching the images to form an ultra-wide-angle photo or video.

[0109] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 8 As shown, the application framework layer may include a window manager, a content provider, a phone manager, a resource manager, a notification manager, a view system, and the like.

[0110] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. In some embodiments of the present application, the application cold start will run in the Android runtime, and the Android runtime will obtain the optimized file status parameters of the application. Then, the Android runtime can determine whether the optimized file is outdated due to system upgrades based on the optimized file status parameters, and return the judgment result to the application management module.

[0111] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0112] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0113] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0114] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0115] The Media Library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The Media Library can support a variety of audio and video encoding formats, such as: MPEG2, H.262, MP3, AAC, AMR, JPG, PNG, etc.

[0116] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing.

[0117] A 2D graphics engine is a drawing engine for 2D drawings.

[0118] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, sensor driver, etc.

[0119] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS and Windows.

Claims

1. A method for removing noise, characterized in that: Applied to an electronic device, the electronic device has an ultra-wide-angle camera, the ultra-wide-angle camera is used to collect multiple frames of images, and the method includes: Acquire a reference frame, where the reference frame is a frame of image selected as a reference from among the multiple frames of image; Determining a first noise reduction intensity based on a noise intensity corresponding to the sensitivity of the ultra-wide-angle camera; Adjusting the first noise reduction intensity based on the light intensity value corresponding to the scene information to obtain a basic noise reduction intensity, wherein the scene information indicates a shooting mode, and the basic noise reduction intensity is an intensity used to remove noise in a central area of ​​the reference frame; According to a rule that the distance between a pixel point of the reference frame and a center point of the reference frame is positively correlated with the noise reduction strength, the basic noise reduction strength is adjusted to obtain the noise reduction strength of the pixel point as the gain mapping of the reference frame; Fusing the gain map with the depth map of the reference frame to obtain a fused image; Based on the fused image, removing noise from the multiple frame images; The multiple frames of images after noise removal are stitched into an ultra-wide-angle image; wherein the noise reduction intensity of the edge area in the ultra-wide-angle image is greater than the noise reduction intensity of the central area.

2. The method according to claim 1, characterized in that: The step of adjusting the basic noise reduction intensity according to a rule that the distance between a pixel point of the reference frame and a center point of the reference frame is positively correlated with the noise reduction intensity, and obtaining the noise reduction intensity of the pixel point as the gain mapping of the reference frame includes: Based on the distance between the pixel point and the center point, the reference frame is divided into a plurality of regions, the plurality of regions comprising a center region and an edge region, the center region being a region composed of pixel points whose distance from the center point is not greater than a first value, the edge region being a region composed of pixel points whose distance from the center point is not less than a second value, and the first value being not greater than the second value; Configuring the noise reduction intensity of the central area to be the basic noise reduction intensity; The noise reduction intensity of the edge area is configured to be a maximum noise reduction intensity, and the maximum noise reduction intensity is greater than the basic noise reduction intensity.

3. The method according to claim 2, characterized in that The multiple regions further include a middle region, which is a region composed of pixel points whose distance from the center point is within a range from the first value to the second value; The method further comprises: adjusting the basic noise reduction strength according to a rule that the distance between the pixel point of the reference frame and the center point of the reference frame is positively correlated with the noise reduction strength, and obtaining the noise reduction strength of the pixel point as the gain mapping of the reference frame; Based on the distance between the edge area and the central area, obtaining a change trend between the basic noise reduction intensity and the maximum noise reduction intensity; Based on the distance between the pixel point in the middle area and the central point and the change trend, the basic noise reduction intensity is adjusted to obtain the noise reduction intensity of the pixel point.

4. The method according to any one of claims 1 to 3, characterized in that: The step of fusing the gain map with the depth map of the reference frame to obtain a fused image includes: The gain map and the depth map of the reference frame are weightedly added to obtain the fused image.

5. The method according to claim 4, characterized in that The removing noise of the multiple frames of images based on the fused image, and splicing the multiple frames of images after the noise is removed into an ultra-wide-angle image, comprises: The fused image and the multi-frame images are input into a model to obtain an ultra-wide-angle image output by the model, wherein the model includes a denoising model, and the denoising model is used to remove noise in the multi-frame images based on the fused image.

6. An electronic device, characterized in that: include: one or more processors, memory, display screens, and cameras; The memory, the display screen, and the camera are coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the noise removal method as described in any one of claims 1-5.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the computer device runs the program, the noise removal method according to any one of claims 1 to 5 is implemented.

Images