Photographing method and electronic device

Abstract

The embodiment of the present application discloses a photographing method and electronic device, which relates to the field of electronic devices, and can avoid the low clarity of the reference frame, thereby avoiding the low clarity of the photographing result obtained by multi-frame fusion, and improving the user experience. The specific scheme is: receiving the user's opening operation of the photographing application; obtaining and storing multiple ZSL frames; receiving the user's shooting operation on the photographing application; obtaining a non-ZSL frame, and determining multiple target ZSL frames; determining the clarity value of the non-ZSL frame and the clarity values ​​of the multiple target ZSL frames, and determining the first target ZSL frame according to the clarity values ​​of the multiple target ZSL frames; when the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame, using the non-ZSL frame as the reference frame, multiple target ZSL frames and ZSL frames are fused through a multi-exposure fusion algorithm to obtain a first fused image frame; based on the first fused image frame, a first photographing result is obtained.

Description

Technical Field

[0001] The present application relates to the field of electronic equipment, and in particular to a photographing method and electronic equipment. Background Art

[0002] With the development of electronic device technology, users have higher and higher requirements for the photo results of electronic devices (such as clarity requirements). In order to meet the user's requirements for photo results, when the user uses the electronic device to take photos, the electronic device can use a multi-exposure fusion algorithm to fuse multiple image frames obtained by the camera of the electronic device, thereby improving the clarity of the photo results to meet the user's requirements for the photo results.

[0003] When an electronic device uses a multi-exposure fusion algorithm to fuse captured image frames, the electronic device first determines the image frame with the highest definition from the multiple image frames acquired by the camera as a reference frame. After that, the other frames in the multiple image frames acquired by the camera are fused with the reference frame as the base frame to obtain the photo result.

[0004] However, since the electronic device uses the reference frame as the base frame to perform multi-frame fusion on other frames among the multiple image frames acquired by the camera, when the clarity of the reference frame is low, the clarity of the photo result obtained by multi-frame fusion will be low, affecting the user experience. Summary of the invention

[0005] The embodiments of the present application provide a photographing method and an electronic device, which can avoid the low clarity of the reference frame, thereby avoiding the low clarity of the photographing result obtained by multi-frame fusion, and improving the user experience.

[0006] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0007] In a first aspect, an embodiment of the present application provides a photographing method, which is applied to an electronic device, which may include a photographing application. The photographing method may include: receiving an opening operation of the photographing application by a user; in response to the opening operation, obtaining and storing multiple zero-delay photographing ZSL frames; receiving a photographing operation of the photographing application by the user; in response to the photographing operation, obtaining a non-ZSL frame and determining multiple target ZSL frames; determining a clarity value of the non-ZSL frame and a clarity value of multiple target ZSL frames, and determining a first target ZSL frame according to the clarity values ​​of the multiple target ZSL frames; the first target frame is a target ZSL frame with the largest clarity value among the multiple target ZSL frames; when the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame, taking the non-ZSL frame as a reference frame, and fusing the multiple target ZSL frames and the non-ZSL frame through a multi-exposure fusion algorithm to obtain a first fused image frame; and obtaining a first photographing result based on the first fused image frame.

[0008] Based on the method described in the first aspect, when determining the reference frame in the multi-exposure fusion process, the ZSL frame (i.e., the reference frame) with the highest definition among multiple ZSL frames is not directly used as the reference frame in the multi-exposure fusion process. Instead, a frame with higher definition is determined as the reference frame among multiple ZSL frames and multiple non-ZSL frames. In this way, the definition of the input frame of the multi-exposure fusion algorithm can be improved, that is, the quality of the input image frame of the multi-exposure fusion algorithm is improved, so that when the multi-exposure fusion algorithm uses the reference frame with higher definition as the reference for multi-frame fusion, the definition of the output frame of the multi-exposure fusion algorithm can be improved, thereby improving the clarity of the photographing result, thereby improving the user experience.

[0009] In combination with the first aspect, in another possible implementation, the above-mentioned non-ZSL frame may include a normal exposure frame, and the above-mentioned non-ZSL frame may be used as a reference frame when the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame. It may include: when the clarity value of the normal exposure frame is greater than the clarity value of the first target ZSL frame, the normal exposure frame is used as the reference frame.

[0010] Based on this possible implementation, the electronic device can determine the reference frame according to the clarity of the normal exposure frame in the non-ZSL frame and the first target ZSL frame. When the clarity value of the normal exposure frame in the non-ZSL frame is greater than the clarity value of the first target ZSL frame, the normal exposure frame in the non-ZSL frame is determined as the reference frame, which can not only reduce the amount of calculation of the reference frame, but also improve the accuracy of determining the reference frame. Thereby, the clarity of the input frame of the multi-exposure fusion algorithm can be further improved.

[0011] In combination with the first aspect, in another possible implementation method, in the case where the clarity value of the normal exposure frame is greater than the clarity value of the first target ZSL frame, taking the normal exposure frame as the reference frame may include: determining a first difference based on the clarity value of the normal exposure frame and the clarity value of the first target ZSL frame; the first difference is the difference between the clarity value of the normal exposure frame and the clarity value of the first target ZSL frame; and when the first difference is greater than the first difference threshold, taking the normal exposure frame as the reference frame.

[0012] Based on this possible implementation, the electronic device can determine the reference frame according to the difference between the clarity value of the normal exposure frame in the non-ZSL frame and the clarity value of the first target ZSL frame. When the difference between the clarity value of the normal exposure frame in the non-ZSL frame and the clarity value of the first target ZSL frame is large, the normal exposure frame in the non-ZSL frame is determined as the reference frame, which can further improve the accuracy of determining the reference frame. Thereby, the clarity of the input frame of the multi-exposure fusion algorithm can be further improved.

[0013] In combination with the first aspect, in another possible implementation, the above-mentioned non-ZSL frame may also include a short exposure frame, the exposure time of the short exposure frame is less than the exposure time of the normal exposure frame, and when the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame, taking the non-ZSL frame as the reference frame may include: determining the hole value of the short exposure frame and the hole value of the first target ZSL frame; determining a second difference according to the hole value of the short exposure frame and the hole value of the first target ZSL frame; the second difference is the difference between the hole value of the short exposure frame and the hole value of the first target ZSL frame; when the second difference is less than the second difference threshold and the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame, taking the normal exposure frame as the reference frame.

[0014] Based on this possible implementation, the electronic device can also determine the reference frame according to the difference between the hole value of the short exposure frame in the non-ZSL frame and the hole value of the first target ZSL frame. When the difference between the hole value of the short exposure frame in the non-ZSL frame and the hole value of the first target ZSL frame is small, the reference frame can be further determined according to the clarity value of the normal exposure frame in the non-ZSL frame and the clarity value of the first target ZSL frame, which can further improve the accuracy of determining the reference frame. Thereby, the clarity of the input frame of the multi-exposure fusion algorithm can be further improved.

[0015] In combination with the first aspect, in another possible implementation method, the above-mentioned non-ZSL frame may also include a long exposure frame, the exposure time of the long exposure frame is greater than the exposure time of the normal exposure frame, and the above-mentioned fusing of multiple target ZSL frames and non-ZSL frames through a multi-exposure fusion algorithm to obtain a first fused image frame may include: determining a clarity value of the long exposure frame; when the clarity value of the long exposure frame is greater than a third threshold, fusing multiple target ZSL frames and non-ZSL frames through a multi-exposure fusion algorithm to obtain a first fused image frame; the non-ZSL frame includes a normal exposure frame, a short exposure frame and a long exposure frame.

[0016] Based on this possible implementation, the electronic device can also determine whether to include the long exposure frame in the fusion process according to the clarity value of the long exposure frame in the non-ZSL frame. When the clarity value of the long exposure frame in the non-ZSL frame is greater than the third threshold, the long exposure frame can be included in the fusion process, thereby further improving the clarity of the input frame of the multi-exposure fusion algorithm.

[0017] In combination with the first aspect, in another possible implementation method, the above-mentioned photographing method may also include: when the clarity value of the normal exposure frame is less than the clarity value of the first target ZSL frame, taking the first target ZSL frame as the reference frame, and fusing multiple target ZSL frames and non-ZSL frames through a multi-exposure fusion algorithm to obtain a second fused image frame; based on the second fused image frame, obtaining a second shooting result.

[0018] Based on this possible implementation method, when the clarity value of the normal exposure frame is less than the clarity value of the first target ZSL frame, the first target ZSL frame is used as the reference frame for fusion, which can improve the clarity of the input frame of the multi-exposure fusion algorithm, that is, the quality of the input image frame of the multi-exposure fusion algorithm is improved. Therefore, when the multi-exposure fusion algorithm performs multi-frame fusion with a reference frame with higher clarity as a reference, the clarity of the output frame of the multi-exposure fusion algorithm can be improved, thereby improving the low clarity of the photographic result and improving the user experience.

[0019] In combination with the first aspect, in another possible implementation, the above-mentioned photographing method may also include: when the second difference is greater than the second difference threshold and the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame, taking the normally exposed frame as the reference frame.

[0020] Based on this possible implementation, when the difference between the hole value of the short exposure frame in the non-ZSL frame and the hole value of the first target ZSL frame is greater than the second difference threshold, the normal exposure frame is used as the reference frame to avoid the fusion ghost problem caused by the excessive difference between the short exposure frame in the non-ZSL frame and the first target ZSL frame. Thus, the clarity of the input frame of the multi-exposure fusion algorithm can be improved.

[0021] In combination with the first aspect, in another possible implementation method, the above-mentioned photographing method may also include: when the clarity value of the long exposure frame is less than a third threshold, fusing multiple target ZSL frames and non-ZSL frames through a multi-exposure fusion algorithm to obtain a third fused image frame; the non-ZSL frame includes a normal exposure frame and a short exposure frame; based on the third fused image frame, obtaining a third shooting result.

[0022] Based on this possible implementation, when the clarity value of the long exposure frame in the non-ZSL frame is less than the third threshold, the long exposure frame may not be involved in the fusion process. Since the exposure time of the long exposure frame is long, it is easy to have the problem of smear and blur. Not involving the long exposure frame in the fusion process can avoid the blur problem introduced by the forced fusion of the long exposure frame, thereby further improving the clarity of the input frame of the multi-exposure fusion algorithm.

[0023] In a second aspect, an embodiment of the present application provides a photographing device, which can be applied to an electronic device to implement the method in the first aspect. The functions of the photographing device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as a receiving module, an acquisition module, a storage module, and a determination module.

[0024] The receiving module may be used to receive a user's operation of opening a camera application.

[0025] The acquisition module can be used to acquire multiple zero-delay photography ZSL frames in response to the opening operation.

[0026] The storage module can be used to store multiple zero-delay photography ZSL frames.

[0027] The receiving module can also be used to receive the user's shooting operation on the camera application.

[0028] The acquisition module can be used to acquire a non-ZSL frame in response to a shooting operation.

[0029] The determination module can be used to determine multiple target ZSL frames.

[0030] The determination module can also be used to determine the clarity value of the non-ZSL frame and the clarity values ​​of multiple target ZSL frames, and determine the first target ZSL frame based on the clarity values ​​of multiple target ZSL frames; the first target frame is the target ZSL frame with the largest clarity value among the multiple target ZSL frames.

[0031] The determination module can also be used to fuse multiple target ZSL frames and non-ZSL frames using a multi-exposure fusion algorithm to obtain a first fused image frame, using the non-ZSL frame as a reference frame when the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame.

[0032] The determination module may also be configured to obtain a first shooting result based on the first fused image frame.

[0033] In a third aspect, a photographing device is provided, which has the function of implementing the method described in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0034] In a fourth aspect, a photographing device is provided, comprising: a processor and a memory; the memory is used to store computer execution instructions, and when the photographing device is running, the processor executes the computer execution instructions stored in the memory, so that the photographing device performs the photographing method as described in any one of the above-mentioned first aspects.

[0035] In a fifth aspect, a photographing device is provided, comprising: a processor; the processor is used to couple with a memory, and after reading instructions in the memory, execute the photographing method as described in any one of the first aspects above according to the instructions.

[0036] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by an electronic device, the electronic device implements the photographing method as described in any one of the first aspect or possible implementations of the first aspect.

[0037] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising a computer-readable code, which, when executed in an electronic device, enables the electronic device to implement a photographing method as described in the first aspect or any possible implementation of the first aspect.

[0038] In an eighth aspect, a device (for example, the device may be a chip system) is provided, the device including a processor for supporting an electronic device to implement the functions involved in the first aspect above. In one possible design, the device also includes a memory for storing program instructions and data necessary for the electronic device. When the device is a chip system, it may be composed of a chip, or may include a chip and other discrete devices.

[0039] It should be understood that the beneficial effects of the second to eighth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the amplitude of hand shaking when a user holds an electronic device to take a photo;

[0041] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of the principle of the photographing method provided in the embodiment of the present application;

[0044] Figure 5 Schematic diagram of the front-end frame selection scheme provided in the embodiment of the present application Figure 1 ;

[0045] Figure 6 Schematic diagram of the front-end frame selection scheme provided in the embodiment of the present application Figure 2 ;

[0046] Figure 7 Schematic diagram of the front-end frame selection scheme provided in the embodiment of the present application Figure 3 ;

[0047] Figure 8 Schematic diagram of the back-end frame selection solution provided in the embodiment of the present application Figure 1 ;

[0048] Fig. 9 Schematic diagram of the back-end frame selection solution provided in the embodiment of the present application Figure 2 ;

[0049] Fig.10 Schematic diagram of the back-end frame selection solution provided in the embodiment of the present application Figure 3 ;

[0050] Fig.11 Schematic diagram of the back-end frame selection solution provided in the embodiment of the present application Figure 4 ;

[0051] Fig.12 A schematic diagram of a process of photographing a method is provided for an embodiment of the present application;

[0052] Fig.13 A schematic diagram of the structure of a photographing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0055] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and appended claims of the present application, the singular expressions "a", "an", "said", "above", "the" and "this" are intended to also include plural expressions, unless the context clearly indicates otherwise. It should also be understood that " / " means or, for example, A / B can mean A or B; "and / or" in the text is merely a description of the association relationship of associated characters, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0056] With the development of electronic device (such as mobile phone, etc.) technology, users have higher and higher requirements (such as clarity) for the photo results (i.e., final image) of electronic devices. In order to meet the user's requirements for photo results, when the user uses the electronic device to take photos, the electronic device can use a multi-exposure fusion algorithm to fuse multiple image frames obtained by the camera of the electronic device, thereby improving the clarity of the photo results to meet the user's requirements for the photo results.

[0057] When an electronic device uses a multi-exposure fusion algorithm to fuse captured image frames, the electronic device first determines the image frame with the highest definition from the multiple image frames acquired by the camera as a reference frame. After that, the other frames in the multiple image frames acquired by the camera are fused with the reference frame as the base frame to obtain the photo result.

[0058] The electronic device determines the image frame with the highest definition from the multiple image frames obtained from the camera as the reference frame, including that after the user opens the camera application included in the electronic device, the electronic device obtains multiple zero shutter lag (ZSL) frames through the camera, and stores the multiple ZSL frames in the cache (i.e., ZSL buffer) of the electronic device, and the format of the ZSL frames is the raw image data (RAW) format. The multiple ZSL frames in the cache of the electronic device are all image frames corresponding to the normal exposure time, that is, the multiple ZSL frames are all normal exposure (normal, N) frames.

[0059] It should be noted that a preset number (such as 8 or 10 frames, etc.) of ZSL frames are usually stored in the cache of the electronic device. Before the user presses the shooting control corresponding to the camera application, that is, before the electronic device receives the user's shooting operation, the electronic device will continue to obtain new ZSL frames through the camera, store new ZSL frames in the cache, and delete the earliest stored ZSL frames. In other words, before the electronic device receives the user's shooting operation, the electronic device dynamically stores new ZSL frames in the cache.

[0060] Afterwards, when the user presses the shooting control corresponding to the photo-taking application, that is, when the electronic device receives the user's shooting operation, the electronic device no longer obtains new ZSL frames through the camera. At this time, the multiple ZSL frames stored in the cache are determined by the electronic device as the final multiple ZSL frames.

[0061] When the electronic device receives a shooting operation from the user, the electronic device acquires multiple non-ZSL frames through the camera. The multiple non-ZSL frames may include image frames corresponding to different exposure times, that is, the multiple non-ZSL frames may include normal exposure frames, short exposure frames, and long exposure frames.

[0062] The non-ZSL frame includes a normal exposure frame, and the image frame corresponding to the normal exposure time in multiple non-ZSL frames (i.e., N frames). The non-ZSL frame includes a short exposure frame, and the image frame corresponding to the short exposure time in multiple non-ZSL frames is a short exposure time (shrot, S) frame. The short exposure time is less than the normal exposure time. The non-ZSL frame includes a long exposure frame, and the image frame corresponding to the short exposure time in multiple non-ZSL frames is a long exposure time (long, L) frame. The long exposure time is greater than the normal exposure time.

[0063] After the electronic device determines the final multiple ZSL frames, the electronic device may respectively determine the clarity values ​​of the multiple ZSL frames, and determine the ZSL frame with the largest clarity value as a reference frame.

[0064] After the electronic device determines the reference frame, the electronic device can use the multi-exposure fusion algorithm to take the reference frame as a benchmark frame, and align and fuse other ZSL frames and non-ZSL frames based on the reference frame, thereby obtaining an output frame corresponding to the multi-exposure fusion algorithm.

[0065] After obtaining the output frame corresponding to the multi-exposure fusion algorithm, the electronic device may further process the output frame corresponding to the multi-exposure fusion algorithm through an image signal processor algorithm to obtain a shooting result.

[0066] That is to say, in the related art, the ZSL frame with the largest clarity value among multiple ZSL frames is used as the reference frame, and the reference frame is used as the basis for alignment and fusion. Therefore, the clarity of the output frame corresponding to the multi-exposure fusion algorithm is affected by the clarity of the reference frame. When the clarity of the reference frame is low (that is, the clarity value of the reference frame is small), the clarity of the output frame corresponding to the multi-exposure fusion algorithm will be low, which will in turn lead to low clarity of the final shooting result.

[0067] When a user holds an electronic device (such as a mobile phone) to take a photo, the user's hand will shake, and the hand shake is random and uncertain, which will cause the clarity of the ZSL frame acquired by the electronic device to be lower than the clarity of the non-ZSL frame. This will result in the output frame corresponding to the multi-exposure fusion algorithm using the ZSL frame as a reference frame and the reference frame as a benchmark for alignment and fusion, resulting in lower clarity, which will in turn lead to lower clarity in the final shooting result.

[0068] Combine the following Figure 1 The randomness and uncertainty of a user's hand shake when the user holds an electronic device (such as a mobile phone) to take a photo is schematically explained.

[0069] Figure 1(a) is the average angular velocity of the user on the X-axis when the user holds the electronic device (such as a mobile phone) to take a photo, that is, the relationship between the magnitude and frequency of the user's shaking amplitude on the X-axis. Figure 1 (b) is the average angular velocity of the user on the Y axis when the user holds an electronic device (such as a mobile phone) to take a photo, that is, the relationship between the magnitude and frequency of the user's shaking amplitude on the Y axis. Figure 1 (a) and Figure 1 From (b), we can see that when a user holds an electronic device (such as a mobile phone) to take a photo, the amplitude of the hand shake mostly oscillates between 0 and 0.5 degrees. When the amplitude of the hand shake is high, it can reach about 1.5 degrees. When the amplitude of the hand shake is small, it can reach about 0.1 degrees. In addition, the frequency of the hand shake is mostly around 25 Hz. Therefore, when a user holds an electronic device (such as a mobile phone) to take a photo, the user's hand shake is random and uncertain.

[0070] Due to the randomness and uncertainty of the user's hand shaking when the user holds the electronic device to shoot, the clarity of the ZSL frame obtained by the electronic device through the camera is lower than the clarity of the non-ZSL frame. As a result, the output frame corresponding to the multi-exposure fusion algorithm using the ZSL frame as the reference frame and the reference frame as the benchmark for alignment and fusion has a lower clarity, which in turn leads to a lower clarity of the final shooting result.

[0071] That is to say, when a user holds an electronic device to take pictures, the randomness and uncertainty of the user's hand shaking will cause the clarity of the input frame of the multi-exposure fusion algorithm to be low, that is, the input RAW of the multi-exposure fusion algorithm is blurred. In particular, when a user holds an electronic device to take night scene photos, due to the long exposure time of night scene photos, the probability of user hand shaking is high during the long exposure time, resulting in low clarity of the input frame of the multi-exposure fusion algorithm. This will in turn cause the clarity of the output frame corresponding to the multi-exposure fusion algorithm to be low, resulting in low clarity of the final shooting result.

[0072] In summary, when an electronic device uses a multi-exposure fusion algorithm to fuse captured image frames, the electronic device first determines the ZSL frame with the highest definition from the multiple ZSL frames acquired by the camera as the reference frame. After that, the other frames in the multiple ZSL frames acquired by the camera and multiple non-ZSL frames are fused with the reference frame as the base frame to obtain the photo result.

[0073] Since the electronic device uses the reference frame as the base frame to perform multi-frame fusion of other frames in the multiple ZSL frames and multiple non-ZSL frames acquired by the camera, when the clarity of the reference frame in the multiple ZSL frames is low (such as the clarity value of the reference frame in the multiple ZSL frames is smaller than the clarity value of the non-ZSL frame), the clarity of the photo result obtained by the multi-frame fusion will be low, affecting the user experience.

[0074] That is to say, since the electronic device uses the reference frame as the base frame to fuse the other frames in the multiple image frames acquired by the camera, when the clarity of the reference frame is low, the clarity of the photo result obtained by the multi-frame fusion will be low, affecting the user experience.

[0075] In response to the above problems, an embodiment of the present application provides a photographing method, which is applied to an electronic device. The photographing method may include that after the electronic device receives an opening operation of a photographing application from a user, the electronic device may acquire and store multiple ZSL frames. After the electronic device receives a shooting operation of a photographing application from a user, the electronic device may acquire multiple non-ZSL frames, and the electronic device may determine the final multiple ZSL frames. Afterwards, the electronic device may respectively determine the clarity value of each ZSL frame in the final multiple ZSL frames, and use the ZSL frame with the largest clarity value as a reference frame. At the same time, the electronic device may determine the clarity value of a normal exposure frame in multiple non-ZSL frames, and when the clarity value of a normal exposure frame in multiple non-ZSL frames is greater than the clarity value of a reference frame, the reference frame is replaced with a normal exposure frame in multiple non-ZSL frames. Afterwards, the electronic device may use the normal exposure frame in multiple non-ZSL frames as a reference frame to perform multi-frame fusion.

[0076] The solution of the present application is to replace the reference frame with the normal exposure frame among multiple non-ZSL frames when the clarity value of the normal exposure frame among multiple non-ZSL frames is greater than the clarity value of the reference frame. That is, the solution of the present application is to determine a frame with higher clarity among multiple ZSL frames and multiple non-ZSL frames as a reference frame. Thereby, the clarity of the input frame of the multi-exposure fusion algorithm can be improved, that is, the quality of the input RAW of the multi-exposure fusion algorithm is improved, so that when the multi-exposure fusion algorithm performs multi-frame fusion with a reference frame with higher clarity as a reference, the clarity of the output frame of the multi-exposure fusion algorithm can be improved, thereby improving the low clarity of the photographing result and improving the user experience.

[0077] The photographing method provided in the embodiment of the present application is described below.

[0078] The photographing method provided in the embodiment of the present application can be applied to the above-mentioned electronic devices. In some embodiments, the above-mentioned electronic devices can be cameras such as mobile phones, tablet computers, handheld computers, personal computers (PCs), cellular phones, personal digital assistants (PDAs), and electronic devices including multi-exposure fusion algorithms. The embodiment of the present application does not limit the specific form of the electronic device.

[0079] For example, taking the electronic device as a mobile phone, Figure 2 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown.

[0080] like Figure 2 As shown, the electronic device may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. Among them, the sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 380F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0081] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0082] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0083] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0084] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 310 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0085] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0086] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor.

[0087] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0088] The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic devices. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 250 can be set in the processor 210. In some embodiments, at least some of the functional modules of the mobile communication module 250 can be set in the same device as at least some of the modules of the processor 210.

[0089] The wireless communication module 260 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to electronic devices. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 210. The wireless communication module 260 can also receive the signal to be sent from the processor 210, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0090] In some embodiments, the antenna 1 of the electronic device is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, so that the electronic device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology.

[0091] The electronic device implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0092] The display screen 394 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 394, where N is a positive integer greater than 1.

[0093] The electronic device can realize the shooting function through the ISP, the camera 293, the video codec, the GPU, the display screen 294 and the application processor, etc. In some embodiments, the electronic device can include 1 or N cameras 293, where N is a positive integer greater than 1.

[0094] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data established during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0095] The acceleration sensor 280E can periodically collect acceleration data of the electronic device at a certain frequency. For example, the acceleration magnitude of the electronic device in various directions (generally XYZ three-axis directions) can be collected.

[0096] Of course, it is understandable that the above Figure 2 The figure is only an exemplary description when the electronic device is in the form of a mobile phone. If the electronic device is in the form of a tablet computer, a handheld computer, a PC, a PDA, a wearable device (such as a smart watch, a smart bracelet), etc., the structure of the electronic device may include Figure 2 The structure shown in the figure is less than Figure 2 More structures are shown in the figure, which are not limited here.

[0097] It is understandable that, generally speaking, the realization of electronic device functions requires not only hardware support but also software cooperation.

[0098] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the electronic device is illustrated.

[0099] Figure 3 It is a software structure block diagram of the electronic device provided in the embodiment of the present application.

[0100] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. The system is divided into four layers, from top to bottom: application layer, application framework layer (also called system framework Java layer), Runtime( runtime) and system libraries (also called Native layer), and kernel layer.

[0101] The application layer can include a series of application packages. Figure 3 As shown, the application package may include photo taking applications (such as camera), gallery, calendar, phone, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0102] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0103] like Figure 3 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0104] like Figure 3 As shown, the application framework layer may also include a camera module.

[0105] The camera module can be used to execute the camera method provided in the embodiment of the present application. That is, the camera module can be used to obtain and store multiple ZSL frames after the electronic device receives the user's operation to open the camera application. The camera module can also be used to obtain multiple non-ZSL frames and determine the final multiple ZSL frames after the electronic device receives the user's operation to shoot the camera application.

[0106] The photographing module can also be used to determine the clarity value of each ZSL frame in the final multiple ZSL frames, and use the ZSL frame with the largest clarity value as a reference frame. The photographing module can also be used to determine the clarity value of a normal exposure frame in multiple non-ZSL frames, and when the clarity value of the normal exposure frame in multiple non-ZSL frames is greater than the clarity value of the reference frame, the reference frame is replaced with a normal exposure frame in multiple non-ZSL frames. The photographing module can also be used to use the normal exposure frame in multiple non-ZSL frames as a reference frame to perform multi-frame fusion.

[0107] Runtime The runtime includes the core library and the virtual machine. Runtime The runtime is responsible for Scheduling and management of the system.

[0108] The core library consists of two parts: one is the function that the Java language needs to call, and the other is The core library.

[0109] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0110] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0111] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0112] The methods in the following embodiments can all be implemented in an electronic device having the above hardware structure or software structure.

[0113] The following is combined with Figure 4 , taking the electronic device as a mobile phone, which may include a camera, and a user taking night scene photos using a mobile phone as an example, the principle of the photo taking method provided in the embodiment of the present application is schematically explained.

[0114] like Figure 4 As shown, after the user opens the camera application included in the mobile phone, the mobile phone obtains multiple ZSL frames through the camera and stores the multiple ZSL frames in the cache of the mobile phone. The multiple ZSL frames are all normal exposure frames (ie, N frames).

[0115] Afterwards, when the user presses the shooting control corresponding to the camera application, that is, when the mobile phone receives the user's shooting operation, the mobile phone no longer obtains new ZSL frames through the camera. At this time, the multiple ZSL frames stored in the cache are determined by the mobile phone as the final multiple ZSL frames.

[0116] When the mobile phone receives the user's shooting operation, the mobile phone will obtain multiple non-ZSL frames through the camera. The multiple non-ZSL frames include image frames corresponding to different exposure times, that is, the multiple non-ZSL frames may include normal exposure frames (i.e., N frames), short exposure frames (i.e., S frames), and long exposure frames (i.e., L frames).

[0117] Afterwards, the mobile phone can perform a front-end frame selection scheme for multiple ZSL frames. That is, the mobile phone can determine the clarity values ​​of the final multiple ZSL frames respectively, and determine the ZSL frame with the highest clarity value as the reference frame corresponding to the multiple ZSL frames. After the mobile phone determines the reference frame from the final multiple ZSL frames, the mobile phone can send the determined reference frame to the multi-exposure fusion algorithm in the mobile phone, and the multi-exposure fusion algorithm performs the back-end frame selection scheme.

[0118] After the mobile phone uses the front-end frame selection scheme to determine the reference frame, the mobile phone's multi-exposure fusion algorithm can determine the final reference frame based on the reference frame and multiple non-ZSL frames acquired by the mobile phone camera. That is, the mobile phone can determine the reference frame in the multi-exposure fusion process based on the clarity values ​​of multiple non-ZSL frames and the clarity values ​​of the reference frame. For example, the mobile phone can compare the clarity value of the normal exposure frame in multiple non-ZSL frames with the clarity value of the reference frame. When the clarity value of the normal exposure frame in multiple non-ZSL frames is greater than the clarity value of the reference frame, the normal exposure frame in multiple non-ZSL frames is used as the reference frame in the multi-exposure fusion process. When the clarity value of the normal exposure frame in multiple non-ZSL frames is less than the clarity value of the reference frame, multiple reference frames are used as the reference frames in the multi-exposure fusion process.

[0119] After the mobile phone determines the reference frame in the multi-exposure fusion process, the mobile phone can use the reference frame as a reference, and align and fuse other ZSL frames and non-ZSL frames based on the reference frame, thereby obtaining an output frame corresponding to the multi-exposure fusion algorithm.

[0120] After the mobile phone obtains the output frame corresponding to the multi-exposure fusion algorithm through the multi-exposure fusion algorithm, the mobile phone can further process the output frame corresponding to the multi-exposure fusion algorithm through the image processing engine (IPE) on the mobile phone. For example, the image processing engine on the mobile phone performs hardware noise reduction, image cropping, noise reduction, color processing, detail enhancement and other image processing on the output frame corresponding to the multi-exposure fusion algorithm, so as to obtain the final night scene imaging.

[0121] The solution of the present application, when determining the reference frame in the multi-exposure fusion process, does not directly use the reference frame corresponding to the multiple ZSL frames (i.e., the ZSL frame with the highest clarity among the multiple ZSL frames) as the reference frame in the multi-exposure fusion process. Instead, when the clarity value of the normal exposure frame among the multiple non-ZSL frames is greater than the clarity value of the reference frame, the reference frame is replaced with the normal exposure frame among the multiple non-ZSL frames, that is, the normal exposure frame among the multiple non-ZSL frames is determined as the reference frame in the multi-exposure fusion process. When the clarity value of the normal exposure frame among the multiple non-ZSL frames is less than the clarity value of the reference frame, the reference frame is determined as the reference frame in the multi-exposure fusion process.

[0122] That is, the solution of the present application is to determine a frame with higher definition as a reference frame among multiple ZSL frames and multiple non-ZSL frames. This can improve the definition of the input frame of the multi-exposure fusion algorithm, that is, improve the quality of the input RAW of the multi-exposure fusion algorithm, so that when the multi-exposure fusion algorithm uses a reference frame with higher definition as a benchmark for multi-frame fusion, it can improve the definition of the output frame of the multi-exposure fusion algorithm, and then improve the clarity of the photo result, thereby improving the user experience.

[0123] It should be noted that when the mobile phone performs a front-end frame selection scheme for multiple ZSL frames, that is, the mobile phone determines the clarity values ​​of the final multiple ZSL frames respectively, and determines the ZSL frame with the highest clarity value as the reference frame corresponding to the multiple ZSL frames, the mobile phone can combine the mobile phone's gyroscope (i.e. gyroscope) information, optical image stabilizer (optical imagestabilization, OIS) information, ZSL frame motion information and ZSL frame ghost detection information and other information to perform clarity calculation, thereby respectively determining the clarity values ​​of the final multiple ZSL frames.

[0124] For example, combined with Figure 5 As shown, for each ZSL frame, the process of the mobile phone determining the definition frame of the ZSL frame is schematically illustrated.

[0125] For each ZSL frame, when the mobile phone determines the definition frame of the ZSL frame, the mobile phone can read the gyroscope data, so as to obtain the rotation information of the mobile phone corresponding to the ZSL frame. In other words, when the mobile phone obtains a ZSL frame, the gyroscope of the mobile phone can record the rotation information of the mobile phone in space corresponding to the ZSL frame. The rotation information is information such as the tilt, roll and movement of the mobile phone in space.

[0126] At the same time, the mobile phone can also obtain the optical image stabilizer data, so as to obtain the image offset information corresponding to the ZSL frame. That is to say, when the mobile phone obtains a ZSL frame, the optical image stabilizer of the mobile phone can record the image offset of the ZSL frame relative to the previous ZSL frame. That is, the mobile phone will use the data of the optical image stabilizer to further analyze the stability of the image. The optical image stabilizer can detect and correct image blur caused by movement or vibration of the mobile phone. By analyzing the optical image stabilizer data, the clarity of the image can be determined.

[0127] The motion information of the ZSL frame refers to the motion variables of the camera corresponding to the ZSL frame calculated by the mobile phone based on the data of the gyroscope and OIS and other sensor information. These variables can include information such as translation, rotation and scaling.

[0128] Ghosting refers to false, unreal shadows that appear in an image. The ghosting detection information of the ZSL frame means that the phone will use image processing algorithms to detect ghosting to identify and correct these shadows. The ghosting detection algorithm may be based on information such as image content, lighting conditions, object shape, etc.

[0129] Afterwards, when the mobile phone determines the clarity frame of the ZSL frame, the mobile phone can combine the rotation information, image offset information, movement variable information and ghost detection information of the mobile phone corresponding to the ZSL frame, and calculate the clarity value of the ZSL frame through a clarity value algorithm.

[0130] After the mobile phone calculates the clarity value of each ZSL frame, the mobile phone can perform front-end frame selection, that is, the mobile phone can sort multiple ZSL frames according to the size of the clarity value, so as to determine the ZSL frame with the highest clarity value as the reference frame corresponding to the multiple ZSL frames.

[0131] It should be noted that when the mobile phone determines the ZSL frame with the highest clarity value as the reference frame corresponding to multiple ZSL frames, the mobile phone can also determine the second clear frame (i.e., the second clear frame) and the third clear frame. The clarity value of the second clear frame is less than the clarity value of the reference frame, and greater than the clarity value of the third clear frame. The clarity value of the third clear frame is greater than the clarity values ​​of other ZSL frames, i.e., the ZSL frames other than the reference frame, the second clear frame, and the third clear frame in the multiple ZSL frames.

[0132] Since the clarity values ​​of other ZSL frames are all less than the clarity value of the third clear frame, the other ZSL frames are relatively blurred frames, and the mobile phone can discard the other ZSL frames. That is, the mobile phone can send the reference frame, the second clear frame and the third clear frame to the multi-exposure fusion algorithm, and the multi-exposure fusion algorithm can perform a back-end frame selection scheme and a multi-exposure fusion process based on the reference frame, the second clear frame and the third clear frame.

[0133] For example, combined with Figure 6 As shown, taking the final multiple ZSL frames determined by the mobile phone including n frames, namely F1 frame, F2 frame, F3 frame, F4 frame, F5 frame and Fn frame as an example, when the mobile phone determines the ZSL frame with the highest clarity value as the reference frame corresponding to multiple ZSL frames, the mobile phone can also determine the second clear frame (ie, the second clear frame) and the third clear frame.

[0134] After the mobile phone calculates the clarity value of each ZSL frame (i.e., the clarity values ​​of the F1 frame, F2 frame, F3 frame, F4 frame, F5 frame and Fn frame), the mobile phone can sort the F1 frame, F2 frame, F3 frame, F4 frame, F5 frame and Fn frame according to the clarity value from large to small.

[0135] When the definition value of the F2 frame is the highest, after reordering, the F2 frame may be located at the first frame (i.e., the F1' frame). When the definition value of the Fn frame is less than the definition value of the F2 frame but greater than the definition values ​​of the F1 frame, the F3 frame, the F4 frame, and the F5 frame, after reordering, the F2 frame may be located at the second frame (i.e., the F2' frame). When the definition value of the F1 frame is less than the definition value of the Fn frame but greater than the definition values ​​of the F3 frame, the F4 frame, and the F5 frame, after reordering, the F2 frame may be located at the third frame (i.e., the F3' frame). When the definition value of the F4 frame is less than the definition value of the F1 frame but greater than the definition values ​​of the F3 frame and the F5 frame, after reordering, the F4 frame may be located at the fourth frame (i.e., the F4' frame). When the definition value of the F3 frame is less than the definition value of the F4 frame but greater than the definition value of the F5 frame, after reordering, the F3 frame may be located at the fifth frame (i.e., the F5' frame). The definition value of the F5 frame is less than that of the F3 frame, and after reordering, the F5 frame can be located at the nth frame (i.e., the Fn' frame). When the definition value of the F1 frame is less than that of the Fn frame, but greater than that of the F3 frame, the F4 frame, and the F5 frame, after reordering, the F2 frame can be located at the third frame (i.e., the F3' frame).

[0136] like Figure 6 As shown, after reordering, the frame with the highest clarity value is F1', followed by F2', F3', F4', F5' and Fn'. Therefore, the mobile phone can determine that F1' is the reference frame, F2' is the second clear frame, and F3' is the third clear frame. Since the clarity values ​​of frames F4', F5' and Fn' are small, F4', F5' and Fn' are blurred, and the mobile phone can discard the F4', F5' and Fn' frames. That is, the mobile phone can send F1', F2' and F3' to the multi-exposure fusion algorithm, and the multi-exposure fusion algorithm can perform a back-end frame selection scheme and a multi-exposure fusion process based on F1', F2' and F3'.

[0137] Combine the following Figure 7 Yes, continuing with the example of the final multiple ZSL frames determined by the mobile phone including n frames, namely F1 frame, F2 frame, F3 frame, F4 frame, F5 frame and Fn frame, the process of the mobile phone discarding F4', frame F5' and frame Fn' when the mobile phone determines the ZSL frame with the highest clarity value as the reference frame corresponding to multiple ZSL frames is schematically explained.

[0138] like Figure 7As shown, when the final multiple ZSL frames determined by the mobile phone include n frames, namely, F1 frame, F2 frame, F3 frame, F4 frame, F5 frame and Fn frame, the mobile phone determines the clarity values ​​of the multiple ZSL frames, and sorts them in descending order according to the clarity values ​​to obtain multiple ZSL frames after reordering, namely, F1' frame to Fn' frame. Among the multiple ZSL frames after reordering, namely, F1' frame has the highest clarity among the multiple ZSL frames from F1' frame to Fn' frame, and is the reference frame. Among the multiple ZSL frames after reordering, namely, F2' frame has the second highest clarity among the multiple ZSL frames from F1' frame to Fn' frame, and is the second clear frame. Among the multiple ZSL frames after reordering, namely, F3' frame has the third highest clarity among the multiple ZSL frames from F1' to Fn', and is the third clear frame. Afterwards, the mobile phone can retain the F1' frame, F2' frame and F3' frame, and discard the F4' frame, F5' frame and Fn' frame.

[0139] After the mobile phone determines the reference frame among multiple ZSL frames (i.e. the ZSL frame with the highest clarity among multiple ZSL frames), the multi-exposure fusion algorithm in the mobile phone can perform a back-end frame selection scheme based on the reference frame and multiple non-ZSL frames acquired by the camera of the mobile phone (multiple non-ZSL frames include normal exposure frames, short exposure frames and long exposure frames).

[0140] Combine the following Figure 8 , a schematic diagram is given of the process of performing a back-end frame selection scheme of the multi-exposure fusion algorithm in the mobile phone according to the reference frame and multiple non-ZSL frames obtained by the camera of the mobile phone.

[0141] like Figure 8 As shown, after the mobile phone determines the reference frame, the mobile phone can send the reference frame to the multi-exposure fusion algorithm in the mobile phone, and after the mobile phone obtains multiple non-ZSL frames through the camera (multiple non-ZSL frames include normal exposure frames, short exposure frames and long exposure frames), the mobile phone can send multiple non-ZSL frames to the multi-exposure fusion algorithm in the mobile phone.

[0142] After the multi-exposure fusion algorithm in the mobile phone obtains the reference frame and the non-ZSL frame (the multiple non-ZSL frames include normal exposure frames, short exposure frames and long exposure frames), the multi-exposure fusion algorithm in the mobile phone can determine whether the clarity value of the normal exposure frame among the multiple non-ZSL frames exceeds the preset threshold of the clarity value of the reference frame.

[0143] After the multi-exposure fusion algorithm in the mobile phone obtains the reference frame and the non-ZSL frame (the multiple non-ZSL frames include normal exposure frames, short exposure frames and long exposure frames), the multi-exposure fusion algorithm in the mobile phone can determine the clarity value of the normal exposure frame in the multiple non-ZSL frames and the clarity value of the reference frame through the clarity operator (such as the sobel operator) in the algorithm. Afterwards, it can be determined whether the multi-exposure fusion algorithm can determine whether the clarity value of the normal exposure frame in the multiple non-ZSL frames exceeds the preset threshold of the clarity value of the reference frame.

[0144] For example, combined with Fig. 9 As shown, after the multi-exposure fusion algorithm in the mobile phone obtains the reference frame, the first clear frame and the second clear frame corresponding to the multiple ZSL frames, the mobile phone can determine the clarity value of the reference frame and the clarity value of the normal exposure frame in the multiple non-ZSL frames through the clarity operator in the multi-exposure fusion algorithm. Afterwards, the multi-exposure fusion algorithm can determine whether the clarity value of the normal exposure frame in the multiple non-ZSL frames exceeds the preset threshold of the clarity value of the reference frame.

[0145] It should be noted that the reference frame, the first clear frame and the second clear frame are all RAW format frames, and the reference frame, the first clear frame and the second clear frame are all image frames corresponding to normal exposure time. Multiple non-ZSL frames are all RAW format frames.

[0146] When the clarity value of the normal exposure frame among multiple non-ZSL frames exceeds the preset threshold of the clarity value of the reference frame, the multi-exposure fusion algorithm in the mobile phone can replace the normal exposure frame with a new reference frame, that is, determine the normal exposure frame as the reference frame in the multi-exposure fusion process.

[0147] After the normal exposure frame is determined as the reference frame in the multi-exposure fusion process, the multi-exposure fusion algorithm in the mobile phone can align and fuse the ZSL frames (i.e., the reference frame, the first clear frame, and the second clear frame) and the non-ZSL frames (i.e., the long exposure frame and the short exposure frame) based on the reference frame, thereby obtaining the output frame corresponding to the multi-exposure fusion algorithm.

[0148] After obtaining the output frame corresponding to the multi-exposure fusion algorithm, the mobile phone can further process the output frame corresponding to the multi-exposure fusion algorithm (such as noise reduction processing). After further processing, the mobile phone can convert the format of the processed output frame corresponding to the multi-exposure fusion algorithm, that is, convert the RAW format to the YUV format. Thus, the mobile phone can output the frame in the YUV format, that is, the final shooting result.

[0149] When the clarity value of the normally exposed frame among multiple non-ZSL frames does not exceed the preset threshold of the clarity value of the reference frame, the multi-exposure fusion algorithm in the mobile phone can determine the reference frame as the benchmark frame in the multi-exposure fusion process.

[0150] After determining the reference frame as the benchmark frame in the multi-exposure fusion process, the multi-exposure fusion algorithm in the mobile phone can align and fuse the ZSL frames (i.e., the first clear frame and the second clear frame) and non-ZSL frames (i.e., normal exposure frames, long exposure frames and short exposure frames) based on the benchmark frame, thereby obtaining the output frame corresponding to the multi-exposure fusion algorithm.

[0151] After obtaining the output frame corresponding to the multi-exposure fusion algorithm, the mobile phone can further process the output frame corresponding to the multi-exposure fusion algorithm (such as noise reduction processing). After further processing, the mobile phone can convert the format of the processed output frame corresponding to the multi-exposure fusion algorithm, that is, convert the RAW format to the YUV format. Thus, the mobile phone can output the frame in the YUV format, that is, the final shooting result.

[0152] It should be noted that, combined with Figure 8 As shown, after the multi-exposure fusion algorithm in the mobile phone obtains the reference frame and the non-ZSL frame (the multiple non-ZSL frames include normal exposure frames, short exposure frames and long exposure frames), the multi-exposure fusion algorithm in the mobile phone can also determine whether the hole value of the short exposure frame in the multiple non-ZSL frames exceeds the preset threshold of the hole value of the reference frame.

[0153] After the multi-exposure fusion algorithm in the mobile phone obtains the reference frame and the non-ZSL frame (multiple non-ZSL frames include normal exposure frames, short exposure frames and long exposure frames), the multi-exposure fusion algorithm in the mobile phone can perform hole detection and ghost detection on the reference frame and the short exposure frame included in the non-ZSL frame, so as to determine the hole value corresponding to the reference frame and the hole value corresponding to the short exposure frame included in the non-ZSL frame. Afterwards, the multi-exposure fusion algorithm can determine whether the hole value of the short exposure frame in the multiple non-ZSL frames exceeds the preset threshold of the hole value of the reference frame.

[0154] For example, combined with Fig.10 As shown, after the multi-exposure fusion algorithm in the mobile phone obtains the reference frame, the first clear frame and the second clear frame corresponding to the multiple ZSL frames, the mobile phone can perform hole detection and ghost detection on the reference frame and the short exposure frame included in the non-ZSL frame through the multi-exposure fusion algorithm, so as to determine the hole value corresponding to the reference frame and the hole value corresponding to the short exposure frame included in the non-ZSL frame. Afterwards, the multi-exposure fusion algorithm can determine whether the hole value of the short exposure frame in the multiple non-ZSL frames exceeds the preset threshold of the hole value of the reference frame.

[0155] When the hole value of the short exposure frame among multiple non-ZSL frames exceeds the preset threshold of the hole value of the reference frame, the multi-exposure fusion algorithm in the mobile phone can replace the normal exposure frame among the multiple non-ZSL frames with a new reference frame, that is, determine the normal exposure frame among the multiple non-ZSL frames as the reference frame in the multi-exposure fusion process.

[0156] After determining the normal exposure frame among multiple non-ZSL frames as the reference frame in the multi-exposure fusion process, the multi-exposure fusion algorithm in the mobile phone can align and fuse the ZSL frames (i.e., the reference frame, the first clear frame, and the second clear frame) and the non-ZSL frames (i.e., the long exposure frame and the short exposure frame) based on the reference frame, thereby obtaining the output frame corresponding to the multi-exposure fusion algorithm.

[0157] After obtaining the output frame corresponding to the multi-exposure fusion algorithm, the mobile phone can further process the output frame corresponding to the multi-exposure fusion algorithm (such as noise reduction processing). After further processing, the mobile phone can convert the format of the processed output frame corresponding to the multi-exposure fusion algorithm, that is, convert the RAW format to the YUV format. Thus, the mobile phone can output the frame in the YUV format, that is, the final shooting result.

[0158] When the hole value of the short exposure frame among multiple non-ZSL frames does not exceed the preset threshold of the hole value of the reference frame, the multi-exposure fusion algorithm in the mobile phone can determine the reference frame as the benchmark frame in the multi-exposure fusion process.

[0159] After determining the reference frame as the benchmark frame in the multi-exposure fusion process, the multi-exposure fusion algorithm in the mobile phone can align and fuse the ZSL frames (i.e., the first clear frame and the second clear frame) and non-ZSL frames (i.e., normal exposure frames, long exposure frames and short exposure frames) based on the benchmark frame, thereby obtaining the output frame corresponding to the multi-exposure fusion algorithm.

[0160] After obtaining the output frame corresponding to the multi-exposure fusion algorithm, the mobile phone can further process the output frame corresponding to the multi-exposure fusion algorithm (such as noise reduction processing). After further processing, the mobile phone can convert the format of the processed output frame corresponding to the multi-exposure fusion algorithm, that is, convert the RAW format to the YUV format. Thus, the mobile phone can output the frame in the YUV format, that is, the final shooting result.

[0161] Since the reference frame is an image frame in multiple ZSL frames, the reference frame is an image frame that has been acquired and stored before the mobile phone camera acquires multiple non-ZSL frames. If the displacement sound changes between the mobile phone camera acquiring the reference frame and acquiring multiple non-ZSL frames, the difference between the reference frame and the short exposure frame in the multiple non-ZSL frames will be too large (that is, the pixel displacement is too large). If the reference frame is still used as the base frame for fusion, the difference between the short exposure frame in the multiple non-ZSL frames and the reference frame is too large, which will cause the fused image to have a ghost problem. Therefore, when the hole value of the short exposure frame in the multiple non-ZSL frames exceeds the preset threshold of the hole value of the reference frame, the multi-exposure fusion algorithm in the mobile phone determines the reference frame as the base frame in the multi-exposure fusion process, which can avoid the difference between the reference frame and the short exposure frame in the multiple non-ZSL frames being too large, thereby avoiding the ghost problem in the fused image.

[0162] It should be noted that when the hole value of the short exposure frame among multiple non-ZSL frames exceeds the preset threshold of the hole value of the reference frame, the multi-exposure fusion algorithm in the mobile phone determines the reference frame as the benchmark frame in the multi-exposure fusion process. Therefore, the multi-exposure fusion algorithm in the mobile phone can first determine whether the hole value of the short exposure frame among multiple non-ZSL frames exceeds the preset threshold of the hole value of the reference frame before determining whether the clarity value of the normal exposure frame among multiple non-ZSL frames exceeds the preset threshold of the clarity value of the reference frame.

[0163] When the hole value of the short exposure frame among multiple non-ZSL frames exceeds the preset threshold of the hole value of the reference frame, the multi-exposure fusion algorithm in the mobile phone determines the reference frame as the benchmark frame in the multi-exposure fusion process. At this time, there is no need to determine whether the clarity value of the normal exposure frame among multiple non-ZSL frames exceeds the preset threshold of the clarity value of the reference frame.

[0164] It should be noted that, combined with Figure 8 As shown, after the multi-exposure fusion algorithm in the mobile phone obtains the reference frame and the non-ZSL frame (the multiple non-ZSL frames include normal exposure frames, short exposure frames and long exposure frames), the multi-exposure fusion algorithm in the mobile phone can also determine whether the clarity value of the long exposure frame among the multiple non-ZSL frames exceeds a preset threshold.

[0165] After the multi-exposure fusion algorithm in the mobile phone obtains the reference frame and the non-ZSL frame (the multiple non-ZSL frames include normal exposure frames, short exposure frames and long exposure frames), the clarity operator in the multi-exposure fusion algorithm in the mobile phone can determine the clarity value of the long exposure frame included in the multiple non-ZSL frames. Afterwards, the multi-exposure fusion algorithm in the mobile phone can determine whether the clarity value of the long exposure frame in the multiple non-ZSL frames exceeds a preset threshold.

[0166] For example, combined with Fig.11As shown, after the multi-exposure fusion algorithm in the mobile phone obtains the reference frame, the first clear frame and the second clear frame corresponding to the multiple ZSL frames, the mobile phone can determine the clarity value of the long exposure frame in the multiple non-ZSL frames through the clarity operator in the multi-exposure fusion algorithm. Afterwards, the multi-exposure fusion algorithm can determine whether the clarity values ​​of the long and short exposure frames of the multiple non-ZSL frames exceed the preset threshold.

[0167] When the clarity value of the long exposure frame among multiple non-ZSL frames exceeds a preset threshold, the multi-exposure fusion algorithm in the mobile phone can determine to continue using the long exposure frame. In other words, the multi-exposure fusion algorithm in the mobile phone can continue to use the long exposure frame among multiple non-ZSL frames to calculate the clarity during the fusion process. For example, the dark part of the fused image frame can use the details of the long exposure frame, thereby improving the clarity of the dark part of the fused image frame. That is, the multi-exposure fusion algorithm in the mobile phone can align and fuse the ZSL frame and the non-ZSL frame (i.e., the normal exposure frame, the long exposure frame, and the short exposure frame) with the reference frame as the reference, thereby obtaining the output frame corresponding to the multi-exposure fusion algorithm.

[0168] After obtaining the output frame corresponding to the multi-exposure fusion algorithm, the mobile phone can further process the output frame corresponding to the multi-exposure fusion algorithm (such as noise reduction processing). After further processing, the mobile phone can convert the format of the processed output frame corresponding to the multi-exposure fusion algorithm, that is, convert the RAW format to the YUV format. Thus, the mobile phone can output the frame in the YUV format, that is, the final shooting result.

[0169] When the clarity value of the long exposure frame in multiple non-ZSL frames does not exceed the preset threshold, the multi-exposure fusion algorithm in the mobile phone can determine to reduce the use or discard the long exposure frame in multiple non-ZSL frames. That is, when the clarity value of the long exposure frame is less than the preset threshold, the usage rate of the long exposure frame is reduced or the long exposure frame is discarded during the fusion process of the multi-exposure fusion algorithm. In other words, the long exposure frame may not participate in the clarity calculation during the fusion process, but the long exposure frame may participate in the brightness calculation during the fusion process. This is because the exposure time of the long exposure frame is long, which is prone to smear and blur problems. If the long exposure frame is forcibly fused, blur problems will be introduced. And the multi-exposure fusion algorithm in the mobile phone can align and fuse the ZSL frame and the non-ZSL frame (that is, the normal exposure frame and the short exposure frame) with the reference frame as the reference, so as to obtain the output frame corresponding to the multi-exposure fusion algorithm.

[0170] After obtaining the output frame corresponding to the multi-exposure fusion algorithm, the mobile phone can further process the output frame corresponding to the multi-exposure fusion algorithm (such as noise reduction processing). After further processing, the mobile phone can convert the format of the processed output frame corresponding to the multi-exposure fusion algorithm, that is, convert the RAW format to the YUV format. Thus, the mobile phone can output the frame in the YUV format, that is, the final shooting result.

[0171] The following is combined with Fig.12 , the photographing method provided in the embodiment of the present application is described in detail, and the photographing method can be applied to electronic devices (such as mobile phones, etc.). Fig.12 As shown, the photographing method may include the following S1201-S1207.

[0172] S1201: The electronic device receives an operation of opening a camera application by a user.

[0173] The electronic device may include a camera application, which may be a built-in camera application of the electronic device or a third-party camera application. The specific type of the camera application is not limited in the embodiments of the present application.

[0174] When a user needs to use an electronic device to take a photo, the user can trigger an icon of a photo application on the electronic device to open the photo application. In other words, the electronic device can receive an opening operation of the photo application by the user. For example, when the electronic device receives a trigger operation of the photo application icon by the user, the electronic device receives the opening operation of the photo application by the user.

[0175] S1202. In response to the opening operation, the electronic device obtains and stores a plurality of zero-delay photo taking ZSL frames.

[0176] When the electronic device receives the user's operation to open the camera application, the electronic device can obtain multiple zero-delay camera ZSL frames in response. For example, when the electronic device receives the user's operation to open the camera application, the electronic device can obtain multiple zero-delay camera ZSL frames through the camera included in the electronic device.

[0177] After the electronic device acquires multiple ZSL frames, the electronic device can store the multiple ZSL frames, such as the electronic device can store the multiple ZSL frames in the cache of the electronic device (i.e., ZSL buffer). The multiple ZSL frames are all in RAW format. The multiple ZSL frames in the cache of the electronic device are all image frames corresponding to normal exposure time, that is, the multiple ZSL frames are all normal exposure (normal, N) frames.

[0178] The exposure time corresponding to the normal exposure frame is the normal exposure time. The normal exposure time of the normal exposure frame refers to the appropriate exposure time used when the exposure time, aperture, sensitivity and other parameters of the camera or camera are reasonably adjusted. The setting of the exposure time directly determines the brightness of the image. Increasing the exposure time can make the image brighter, while reducing the exposure time will make the image darker.

[0179] It should be noted that a preset number (such as 8 or 10 frames) of ZSL frames are usually stored in the cache of the electronic device. Before the electronic device receives the user's shooting operation, the electronic device will continue to obtain new ZSL frames through the camera, store new ZSL frames in the cache, and delete the earliest stored ZSL frames. In other words, before the electronic device receives the user's shooting operation, the electronic device dynamically stores new ZSL frames in the cache.

[0180] S1203: The electronic device receives a shooting operation of a user on a camera application.

[0181] After the user uses the electronic device to open the camera application, the user can use the electronic device to take a photo. That is, the electronic device can receive the user's shooting operation on the camera application. For example, when the electronic device receives the user's trigger operation on the shooting control corresponding to the camera application, the electronic device receives the user's shooting operation on the camera application.

[0182] S1204: In response to the shooting operation, the electronic device acquires a non-ZSL frame and determines a plurality of target ZSL frames.

[0183] When the electronic device receives a user's shooting operation on a camera application, the electronic device can obtain multiple non-ZSL frames in response. For example, when the electronic device receives a user's shooting operation on a camera application, the electronic device can obtain non-ZSL frames through the camera of the electronic device.

[0184] In some examples, the non-ZSL frame acquired by the electronic device may be one or more. The specific number of non-ZSL frames acquired by the electronic device is not limited in the embodiment of the present application. The embodiment of the present application is illustrated by taking the non-ZSL frame acquired by the electronic device as an example.

[0185] In some examples, when the electronic device acquires multiple non-ZSL frames, the non-ZSL frames may include normal exposure frames, short exposure frames, and long exposure frames. The exposure time of the short exposure frame is shorter than the exposure time of the normal exposure frame, and the exposure time of the long exposure frame is longer than the exposure time of the normal exposure frame.

[0186] It should be noted that when a non-ZSL frame may include a normal exposure frame, a short exposure frame, and a long exposure frame, the normal exposure frame may be one or more, and the specific number of normal exposure frames is not limited in the embodiment of the present application. The embodiment of the present application is illustrated by taking one normal exposure frame as an example. The short exposure frame may be one or more, and the specific number of short exposure frames is not limited in the embodiment of the present application. The embodiment of the present application is illustrated by taking one short exposure frame as an example. The long exposure frame may be one or more, and the specific number of long exposure frames is not limited in the embodiment of the present application. The embodiment of the present application is illustrated by taking one long exposure frame as an example.

[0187] When the electronic device receives a shooting operation of a user on a camera application, in response, the electronic device may determine multiple target ZSL frames while acquiring multiple non-ZSL frames.

[0188] The target ZSL frame means that when the electronic device receives the shooting operation of the user's camera application, the electronic device no longer obtains a new ZSL frame through the camera. At this time, the electronic device determines the multiple ZSL frames stored in the cache as the final multiple ZSL frames, that is, the target ZSL frame.

[0189] S1205. The electronic device determines a definition value of the non-ZSL frame and definition values ​​of multiple target ZSL frames, and determines a first target ZSL frame according to the definition values ​​of the multiple target ZSL frames.

[0190] After the electronic device acquires the non-ZSL frame and determines a plurality of target ZSL frames, the electronic device may determine a definition value of the non-ZSL frame and definition values ​​of the plurality of target ZSL frames.

[0191] In some examples, when the clarity value of each target ZSL frame in multiple target ZSL frames of the electronic device is obtained, the electronic device can obtain it through a clarity algorithm. The specific type of the clarity algorithm is not limited in the embodiments of the present application, and it is sufficient to determine the clarity value of the target ZSL frame. For example, the clarity algorithm may include a sharpness algorithm, a contrast algorithm, a statistical feature algorithm, etc.

[0192] In other examples, when determining the clarity value of each target ZSL frame among multiple target ZSL frames of an electronic device, the electronic device may determine the clarity value of the target ZSL frame by combining physical information corresponding to the target ZSL frame and image information of the target ZSL frame.

[0193] The physical information corresponding to the target ZSL frame may include at least one of the gyroscope information of the electronic device, the optical image stabilizer information, and the movement variable of the target ZSL frame when the camera of the electronic device acquires the target ZSL frame. The movement variable of the target ZSL frame is illustrated in the embodiment of the present application by taking the example that the physical information corresponding to the target ZSL frame includes the gyroscope information of the electronic device, the optical image stabilizer information, and the movement variable of the target ZSL frame when the camera of the electronic device acquires the target ZSL frame.

[0194] The gyroscope information of the electronic device may include the rotation information of the electronic device recorded by the gyroscope of the electronic device when the camera of the electronic device acquires the target ZSL frame, such as the tilt, roll and movement of the electronic device in space. The optical image stabilizer information of the electronic device may include the image offset information of the ZSL frame relative to the previous ZSL frame recorded by the optical image stabilizer of the electronic device when the camera of the electronic device acquires the target ZSL frame. The movement variable information of the target ZSL frame may include the movement variables of the camera corresponding to the target ZSL frame calculated by the electronic device based on the information of the gyroscope and the optical image stabilizer and other sensor information, such as the variables may include information such as translation, rotation and scaling.

[0195] The image information of the target ZSL frame may include information obtained by the electronic device through ghost detection on the target ZSL frame. For example, the electronic device may perform ghost detection on the target ZSL frame through a ghost detection algorithm, thereby obtaining the image information of the target ZSL frame, that is, the ghost information. In the embodiments of the present application, the specific type of the ghost detection algorithm is not limited, and it is sufficient that the image information of the target ZSL frame can be obtained. For example, the ghost detection algorithm may include an algorithm based on frame difference, an algorithm based on optical flow, an algorithm based on texture features, and an algorithm based on neural network.

[0196] That is to say, when determining the clarity value of each target ZSL frame among multiple target ZSL frames of an electronic device, the electronic device can first obtain the physical information corresponding to the target ZSL frame (such as gyroscope information, optical image stabilizer information, and movement variables of the target ZSL frame) and the image information of the target ZSL frame (such as ghost detection information). Afterwards, the clarity value of the target ZSL frame is determined based on the physical information corresponding to the target ZSL frame and the image information of the target ZSL frame through a clarity algorithm. Since the clarity value of the target ZSL frame is determined in combination with the physical information corresponding to the target ZSL frame and the image information of the target ZSL frame, the clarity value of the target ZSL frame that can be determined is more accurate.

[0197] After the electronic device determines the clarity value of each target ZSL frame in the multiple target ZSL frames, the electronic device can determine the first target ZSL frame based on the clarity values ​​of the multiple target ZSL frames. For example, the electronic device can sort the clarity values ​​of the multiple target ZSL frames in descending order, thereby determining the target ZSL frame with the largest clarity value as the first target ZSL frame.

[0198] It should be noted that when determining the first target ZSL frame based on the clarity values ​​of multiple target ZSL frames, the electronic device can also determine the second clear frame and the third clear frame. The second clear frame is a target ZSL frame whose clarity value among the multiple target ZSL frames is less than the clarity value of the first target ZSL frame, and is greater than the clarity values ​​of other target ZSL frames (that is, the target ZSL frames other than the first target ZSL frame among the multiple target ZSL frames). The third clear frame is a target ZSL frame whose clarity value among the multiple target ZSL frames is less than the clarity value of the first target ZSL frame and the clarity value of the second clear frame, and is greater than the clarity values ​​of other target ZSL frames (that is, the target ZSL frames other than the first target ZSL frame and the second clear frame among the multiple target ZSL frames).

[0199] After the electronic device determines the first target ZSL frame, the second clear frame and the third clear frame, the electronic device may discard other target ZSL frames (i.e., target ZSL frames other than the first target ZSL frame, the second clear frame and the third clear frame among the multiple target ZSL frames). That is, other target ZSL frames do not participate in the subsequent fusion process.

[0200] After the electronic device determines the first target ZSL frame, the second clear frame, and the third clear frame, the electronic device may send the first target ZSL frame, the second clear frame, and the third clear frame to the multi-exposure fusion algorithm in the electronic device. Thus, the multi-exposure fusion algorithm in the electronic device may determine the reference frame in the multi-exposure fusion process based on the first target ZSL frame, and perform the multi-exposure fusion process based on the first target ZSL frame, the second clear frame, and the third clear frame.

[0201] S1206. When the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame, the electronic device uses the non-ZSL frame as a reference frame, and fuses multiple target ZSL frames and non-ZSL frames through a multi-exposure fusion algorithm to obtain a first fused image frame.

[0202] After the electronic device determines the clarity value of the non-ZSL frame and the first target ZSL frame, the electronic device can determine whether the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame. For example, after the electronic device acquires the non-ZSL frame and determines the first target ZSL frame, the clarity operator (such as the sobel operator) in the multi-exposure fusion algorithm of the electronic device can determine the clarity value of the non-ZSL frame and the clarity value of the first target ZSL frame, and determine whether the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame.

[0203] When the electronic device determines that the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame, the electronic device can use the non-ZSL frame as a reference frame, and fuse multiple target ZSL frames (such as the first target ZSL frame, the second clear frame, and the third clear frame) and non-ZSL frames (such as the normal exposure frame, the short exposure frame, and the long exposure frame) through a multi-exposure fusion algorithm to obtain a first fused image frame. The format of the first fused image frame can be a RAW format. Afterwards, the electronic device can obtain a first shooting result based on the first fused image frame.

[0204] The multi-exposure fusion algorithm is used to process underexposed or overexposed images by fusing multiple images with different exposure levels together to obtain better details and color balance. The specific type of the multi-exposure fusion algorithm is not limited in the embodiment of the present application, and it is sufficient to fuse multiple target ZSL frames and non-ZSL frames to obtain a first fused image frame. For example, the multi-exposure fusion algorithm may include a weighted average algorithm, a lens distortion algorithm, and an exposure fusion algorithm.

[0205] In some examples, after the electronic device obtains the first fused image frame, the electronic device can process the first fused image frame (such as noise reduction processing), so that the electronic device can send the processed first fused image frame to the image processing engine IPE of the electronic device for further image processing, so as to obtain a photo result.

[0206] When the electronic device determines that the clarity value of the non-ZSL frame is less than the clarity value of the first target ZSL frame, the electronic device can use the first target ZSL frame as a reference frame, and fuse multiple target ZSL frames (such as the first target ZSL frame, the second clear frame, and the third clear frame) and non-ZSL frames (such as the normal exposure frame, the short exposure frame, and the long exposure frame) through a multi-exposure fusion algorithm to obtain a second fused image frame. The format of the second fused image frame can be a RAW format. Afterwards, the electronic device can obtain a second shooting result based on the second fused image frame.

[0207] In some examples, after the electronic device obtains the second fused image, the electronic device can process the second fused image (such as noise reduction processing), so that the electronic device can send the processed image to the image processing engine IPE of the electronic device for further image processing, thereby obtaining a photo result.

[0208] It should be noted that when the clarity value of the non-ZSL frame is equal to the clarity value of the first target ZSL frame, the electronic device can use the non-ZSL frame as the reference frame, or the electronic device can use the first target ZSL frame as the reference frame, which is not limited in the embodiments of the present application.

[0209] In some examples, when the non-ZSL frame includes a normal exposure frame, the above-mentioned use of the non-ZSL frame as a reference frame when the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame may include: when the clarity value of the normal exposure frame is greater than the clarity value of the first target ZSL frame, the electronic device may use the normal exposure frame as a reference frame.

[0210] When the clarity value of the normal exposure frame is less than the clarity value of the first target ZSL frame, the electronic device may use the first target ZSL frame as a reference frame.

[0211] It should be noted that when the clarity value of the normal exposure frame is equal to the clarity value of the first target ZSL frame, the electronic device can use the normal exposure frame as the reference frame, and the electronic device can also use the first target ZSL frame as the reference frame. This is not limited in the embodiments of the present application.

[0212] In some examples, when the non-ZSL frame includes a normal exposure frame, and the clarity value of the normal exposure frame is greater than the clarity value of the first target ZSL frame, taking the normal exposure frame as the reference frame may include: the electronic device determines a first difference value according to the clarity value of the normal exposure frame and the clarity value of the first target ZSL frame. The first difference value is the difference between the clarity value of the normal exposure frame and the clarity value of the first target ZSL frame.

[0213] When the first difference is greater than the first difference threshold, the electronic device may use the normal exposure frame as the reference frame. When the first difference is less than the first difference threshold, the electronic device may use the first target ZSL frame as the reference frame.

[0214] The specific value of the first difference threshold can be set according to actual conditions, and the specific value of the first difference threshold is not limited in the embodiment of the present application.

[0215] It should be noted that when the first difference is equal to the first difference threshold, the electronic device can use the normal exposure frame as the reference frame, and the electronic device can also use the first target ZSL frame as the reference frame, which is not limited in the embodiments of the present application.

[0216] In some examples, when the non-ZSL frame includes multiple normal exposure frames, the electronic device may determine the normal exposure frame with the largest clarity value based on the clarity values ​​of the multiple normal exposure frames. Afterwards, the electronic device may compare the clarity value of the normal exposure frame with the largest clarity value with the clarity value of the first target ZSL frame. When the clarity value of the normal exposure frame with the largest clarity value is greater than the clarity value of the first target ZSL frame, the electronic device uses the normal exposure frame with the largest clarity value as the reference frame.

[0217] When the clarity value of the normal exposure frame with the maximum clarity value is smaller than the clarity value of the first target ZSL frame, the electronic device uses the first target ZSL frame as a reference frame.

[0218] It should be noted that when the clarity value of the normal exposure frame with the largest clarity value is equal to the clarity value of the first target ZSL frame, the electronic device can use the normal exposure frame with the largest clarity value as the reference frame, and the electronic device can also use the first target ZSL frame as the reference frame. This is not limited in the embodiments of the present application.

[0219] In some examples, the non-ZSL frame may also include a short exposure frame, the exposure time of the short exposure frame is less than the exposure time of the normal exposure frame, and in the case where the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame, taking the non-ZSL frame as the reference frame may include: the electronic device determines the hole value of the short exposure frame and the hole value of the first target ZSL frame. Afterwards, the electronic device may determine the second difference value based on the hole value of the short exposure frame and the hole value of the first target ZSL frame. The second difference value may be the difference between the hole value of the short exposure frame and the hole value of the first target ZSL frame.

[0220] When the second difference is less than the second difference threshold, the electronic device can continue to determine the reference frame according to the clarity value of the non-ZSL frame and the clarity value of the first target ZSL frame. That is, when the second difference is less than the second difference threshold and the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame, the electronic device can use the normal exposure frame as the reference frame.

[0221] When the second difference is less than the second difference threshold and the definition value of the non-ZSL frame is less than the definition value of the first target ZSL frame, the electronic device may use the first target ZSL as a reference frame.

[0222] When the second difference is greater than the second difference threshold, the electronic device can use the normal exposure frame as the reference frame. That is, when the second difference threshold is greater than the second difference threshold, the electronic device can determine the normal exposure frame as the reference frame, without continuing to determine the reference frame based on the clarity value of the non-ZSL frame and the clarity value of the first target ZSL frame.

[0223] It should be noted that the specific value of the second difference threshold can be set according to actual conditions, and the specific value of the second difference threshold is not limited in the embodiment of the present application. When the second difference is equal to the second difference threshold, the electronic device can use the normal exposure frame as the reference frame, and the electronic device can also use the first target ZSL frame as the reference frame, which is not limited in the embodiment of the present application.

[0224] It should be noted that when the non-ZSL frame includes multiple short exposure frames, the electronic device can determine the hole value of the first short exposure frame among the multiple short exposure frames, and determine the second difference based on the hole value of the first short exposure frame and the hole value of the first target ZSL frame.

[0225] In some examples, when the non-ZSL frame also includes a long exposure frame, the exposure time of the long exposure frame is greater than the exposure time of the normal exposure frame, and the above-mentioned fusion of multiple target ZSL frames and non-ZSL frames through a multi-exposure fusion algorithm to obtain a first fused image frame may include: the electronic device may determine the clarity value of the long exposure frame. Afterwards, the electronic device may fuse the multiple target ZSL frames and non-ZSL frames through a multi-exposure fusion algorithm when the clarity value of the long exposure frame is greater than a third threshold to obtain the first fused image frame. The non-ZSL frame may include a normal exposure frame, a short exposure frame, and a long exposure frame.

[0226] The specific value of the third threshold can be set according to actual conditions, and the specific value of the third threshold is not limited in the embodiments of the present application.

[0227] When the sharpness value of the long exposure frame is less than the third threshold, multiple target ZSL frames and non-ZSL frames are fused by the multi-exposure fusion algorithm to obtain a third fused image frame. The ZSL frame may include a normal exposure frame and a short exposure frame. That is, when the sharpness value of the long exposure frame is less than the third threshold, the usage rate of the long exposure frame is reduced or the long exposure frame is discarded during the fusion process of the multi-exposure fusion algorithm, that is, the long exposure frame may not participate in the sharpness calculation during the fusion process, but the long exposure frame may participate in the brightness calculation during the fusion process.

[0228] It should be noted that when the non-ZSL frame includes multiple long exposure frames, the electronic device can determine the long exposure frame with the highest clarity value among the multiple long exposure frames. Thereafter, the electronic device can determine whether the clarity value of the long exposure frame with the highest clarity value is greater than the third threshold.

[0229] S1207: The electronic device obtains a first shooting result based on the first fused image frame.

[0230] After the electronic device obtains the first fused image frame, the electronic device may obtain a first shooting result based on the first fused image frame.

[0231] For example, after the electronic device obtains the first fused image frame (the format of the first fused image frame is RAW format), the electronic device may perform noise reduction processing on the first fused image frame to obtain the first fused image frame after noise reduction processing (the format of the first fused image frame after noise reduction processing is RAW format). Afterwards, the electronic device may convert the format of the first fused image frame, that is, convert the format of the first fused image frame after noise reduction processing from RAW format to YUV format, thereby obtaining the first shooting result.

[0232] After the electronic device obtains the first shooting result in the YUV format, the electronic device may also display the first shooting result in the YUV format.

[0233] The solution of the present application, when determining the reference frame in the multi-exposure fusion process, does not directly use the ZSL frame (i.e., reference frame) with the highest definition among multiple ZSL frames as the reference frame in the multi-exposure fusion process. Instead, a frame with higher definition is determined as the reference frame among multiple ZSL frames and multiple non-ZSL frames. This can improve the definition of the input frame of the multi-exposure fusion algorithm, that is, improve the quality of the input image frame of the multi-exposure fusion algorithm, so that when the multi-exposure fusion algorithm uses the reference frame with higher definition as the reference for multi-frame fusion, it can improve the definition of the output frame of the multi-exposure fusion algorithm, thereby improving the low definition of the photographing result and improving the user experience.

[0234] Corresponding to the method in the above embodiment, the embodiment of the present application further provides a photographing device. The photographing device can be applied to an electronic device to implement the method in the above embodiment. The function of the photographing device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0235] For example, Fig.13 A schematic diagram of the structure of a photographing device 13 is shown. Fig.13 As shown, the control device 13 may include: a receiving module 1301, an acquisition module 1302, a storage module 1303, a determination module 1304, etc.

[0236] The receiving module 1301 may be used to receive a user's operation of opening a camera application.

[0237] The acquisition module 1302 can be used to acquire multiple zero-delay shooting ZSL frames in response to the opening operation.

[0238] The storage module 1303 can be used to store multiple zero-delay photography ZSL frames.

[0239] The receiving module 1301 may also be used to receive a user's shooting operation on a camera application.

[0240] The acquisition module 1302 may be configured to acquire a non-ZSL frame in response to a capture operation.

[0241] The determination module 1304 may be configured to determine a plurality of target ZSL frames.

[0242] Determination module 1304 can also be used to determine the clarity value of a non-ZSL frame and the clarity values ​​of multiple target ZSL frames, and determine a first target ZSL frame based on the clarity values ​​of multiple target ZSL frames; the first target frame is the target ZSL frame with the largest clarity value among the multiple target ZSL frames.

[0243] Determination module 1304 can also be used to fuse multiple target ZSL frames and non-ZSL frames using a multi-exposure fusion algorithm with the non-ZSL frame as the reference frame to obtain a first fused image frame when the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame.

[0244] The determination module 1304 may also be configured to obtain a first shooting result based on the first fused image frame.

[0245] In another possible implementation, the determination module 1304 may also be configured to use the normal exposure frame as a reference frame when the clarity value of the normal exposure frame is greater than the clarity value of the first target ZSL frame.

[0246] In another possible implementation, the determination module 1304 can also be used to determine a first difference based on the clarity value of the normal exposure frame and the clarity value of the first target ZSL frame; the first difference is the difference between the clarity value of the normal exposure frame and the clarity value of the first target ZSL frame.

[0247] The determination module 1304 may also be configured to use the normally exposed frame as a reference frame when the first difference is greater than a first difference threshold.

[0248] In another possible implementation, the determination module 1304 may also be used to determine the hole value of the short exposure frame and the hole value of the first target ZSL frame.

[0249] The determination module 1304 may also be used to determine a second difference value according to the hole value of the short exposure frame and the hole value of the first target ZSL frame; the second difference value is the difference between the hole value of the short exposure frame and the hole value of the first target ZSL frame;

[0250] The determination module 1304 may also be configured to use the normal exposure frame as the reference frame when the second difference is less than a second difference threshold and the definition value of the non-ZSL frame is greater than the definition value of the first target ZSL frame.

[0251] In another possible implementation, the determination module 1304 may also be used to determine a definition value of the long exposure frame.

[0252] Determination module 1304 can also be used to fuse multiple target ZSL frames and non-ZSL frames through a multi-exposure fusion algorithm to obtain a first fused image frame when the clarity value of the long exposure frame is greater than a third threshold; the non-ZSL frame includes a normal exposure frame, a short exposure frame and a long exposure frame.

[0253] In another possible implementation, the determination module 1304 can also be used to, when the clarity value of the normal exposure frame is less than the clarity value of the first target ZSL frame, use the first target ZSL frame as a reference frame, and fuse multiple target ZSL frames and non-ZSL frames through a multi-exposure fusion algorithm to obtain a second fused image frame.

[0254] The determination module 1304 may also be configured to obtain a second shooting result based on the second fused image frame.

[0255] In another possible implementation, the determination module 1304 may also be used to use the normal exposure frame as the reference frame when the second difference is greater than the second difference threshold and the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame.

[0256] In another possible implementation, the determination module 1304 can also be used to fuse multiple target ZSL frames and non-ZSL frames through a multi-exposure fusion algorithm to obtain a third fused image frame when the clarity value of the long exposure frame is less than a third threshold; the ZSL frame includes a normal exposure frame and a short exposure frame.

[0257] The determination module 1304 may also be configured to obtain a third shooting result based on the third fused image frame.

[0258] It should be understood that the division of units or modules (hereinafter referred to as units) in the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or physically separated. And the units in the device can all be implemented in the form of software calling through processing elements; or all in the form of hardware; or some units can be implemented in the form of software calling through processing elements, and some units can be implemented in the form of hardware.

[0259] For example, each unit can be a separately established processing element, or it can be integrated in a certain chip of the device. In addition, it can also be stored in a memory in the form of a program, and called and executed by a certain processing element of the device. In addition, all or part of these units can be integrated together, or they can be implemented independently. The processing element described here can also be called a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each unit above can be implemented by an integrated logic circuit of hardware in the processor element or in the form of software called by a processing element.

[0260] In one example, the unit in the above apparatus may be one or more integrated circuits configured to implement the above method, such as one or more ASICs, or one or more DSPs, or one or more FPGAs, or a combination of at least two of these integrated circuit forms.

[0261] For another example, when the units in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a CPU or other processor that can call a program. For another example, these units can be integrated together and implemented in the form of a system on a chip SOC.

[0262] In one implementation, the units of the above apparatus implementing the corresponding steps in the above method can be implemented in the form of a processing element scheduling program. For example, the apparatus may include a processing element and a storage element, and the processing element calls the program stored in the storage element to execute the method described in the above method embodiment. The storage element may be a storage element on the same chip as the processing element, that is, an on-chip storage element.

[0263] In another implementation, the program for executing the above method may be in a storage element on a different chip from the processing element, i.e., an off-chip storage element. In this case, the processing element calls or displays the program on the on-chip storage element from the off-chip storage element to call and execute the method described in the above method embodiment.

[0264] For example, an embodiment of the present application may also provide a device, such as an electronic device, which may include a processor and a memory for storing instructions executable by the processor. When the processor is configured to execute the above instructions, the electronic device implements the photographing method described in the above embodiment. The memory may be located inside the electronic device or outside the electronic device. And the processor includes one or more.

[0265] In another implementation, the unit of the device implementing each step in the above method may be configured as one or more processing elements, which may be arranged on the corresponding electronic device, and the processing element here may be an integrated circuit, for example: one or more ASICs, or one or more DSPs, or one or more FPGAs, or a combination of these integrated circuits. These integrated circuits may be integrated together to form a chip.

[0266] For example, the present application also provides a chip, which can be applied to the above electronic device. The chip includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through a line; the processor receives and executes computer instructions from the memory of the electronic device through the interface circuit to implement the method described in the above method embodiment.

[0267] An embodiment of the present application also provides a computer program product, including computer instructions executed by the above-mentioned electronic device.

[0268] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0269] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0270] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0271] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0272] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, such as a program. The software product is stored in a program product, such as a computer-readable storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0273] For example, the embodiment of the present application may also provide a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by an electronic device, the electronic device implements the photographing method described in the aforementioned method embodiment.

[0274] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A photographing method, characterized in that: Applied to an electronic device, the electronic device includes a photo taking application, and the method includes: Receiving an opening operation of the camera application from a user; In response to the opening operation, acquiring and storing a plurality of zero-delay photography ZSL frames; Receiving a shooting operation of the user on the camera application; In response to the shooting operation, a non-ZSL frame is acquired, and a plurality of currently stored ZSL frames are determined as a plurality of target ZSL frames; Determine the definition value of the non-ZSL frame and the definition values ​​of the multiple target ZSL frames, and determine a first target ZSL frame according to the definition values ​​of the multiple target ZSL frames; the first target frame is a target ZSL frame having the largest definition value among the multiple target ZSL frames; In the case where the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame, the non-ZSL frame is used as a reference frame, and the multiple target ZSL frames and the non-ZSL frame are fused by a multi-exposure fusion algorithm to obtain a first fused image frame; the exposure time of the target ZSL frame is a normal exposure time; the non-ZSL frame includes a plurality of image frames with different exposure times, and the plurality of image frames with different exposure times includes an image frame with an exposure time of the normal exposure time; A first shooting result is obtained based on the first fused image frame.

2. The method according to claim 1, characterized in that: The non-ZSL frame includes a normal exposure frame, and when the definition value of the non-ZSL frame is greater than the definition value of the first target ZSL frame, taking the non-ZSL frame as a reference frame includes: In a case where the clarity value of the normal exposure frame is greater than the clarity value of the first target ZSL frame, the normal exposure frame is used as the reference frame.

3. The method according to claim 2, characterized in that In a case where the definition value of the normal exposure frame is greater than the definition value of the first target ZSL frame, taking the normal exposure frame as the reference frame comprises: Determine a first difference value according to the clarity value of the normal exposure frame and the clarity value of the first target ZSL frame; the first difference value is the difference between the clarity value of the normal exposure frame and the clarity value of the first target ZSL frame; When the first difference is greater than a first difference threshold, the normally exposed frame is used as a reference frame.

4. The method according to claim 3, characterized in that The non-ZSL frame further includes a short exposure frame, the exposure time of the short exposure frame is shorter than the exposure time of the normal exposure frame, and when the definition value of the non-ZSL frame is greater than the definition value of the first target ZSL frame, the non-ZSL frame is used as a reference frame, including: Determining a hole value of the short exposure frame and a hole value of the first target ZSL frame; Determine a second difference value according to the hole value of the short exposure frame and the hole value of the first target ZSL frame; the second difference value is the difference between the hole value of the short exposure frame and the hole value of the first target ZSL frame; When the second difference is smaller than a second difference threshold and the definition value of the non-ZSL frame is greater than the definition value of the first target ZSL frame, the normally exposed frame is used as the reference frame.

5. The method according to claim 4, characterized in that The non-ZSL frame also includes a long exposure frame, the exposure time of the long exposure frame is greater than the exposure time of the normal exposure frame, and the multiple target ZSL frames and the non-ZSL frames are fused by a multi-exposure fusion algorithm to obtain a first fused image frame, including: Determining a clarity value of the long exposure frame; When the clarity value of the long exposure frame is greater than a third threshold, the multiple target ZSL frames and the non-ZSL frames are fused by the multi-exposure fusion algorithm to obtain the first fused image frame; the non-ZSL frame includes the normal exposure frame, the short exposure frame and the long exposure frame.

6. The method according to claim 2, characterized in that The method further comprises: In a case where the clarity value of the normal exposure frame is less than the clarity value of the first target ZSL frame, taking the first target ZSL frame as the reference frame, and fusing the multiple target ZSL frames and the non-ZSL frame through a multi-exposure fusion algorithm to obtain a second fused image frame; A second shooting result is obtained based on the second fused image frame.

7. The method according to claim 4, characterized in that The method further comprises: When the second difference is greater than the second difference threshold and the clarity value of the non-ZSL frame is greater than the clarity value of the first target ZSL frame, the normally exposed frame is used as the reference frame.

8. The method according to claim 5, characterized in that The method further comprises: When the definition value of the long exposure frame is less than a third threshold, the multiple target ZSL frames and the non-ZSL frames are fused by the multi-exposure fusion algorithm to obtain a third fused image frame; the ZSL frame includes the normal exposure frame and the short exposure frame; Based on the third fused image frame, a third shooting result is obtained.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory for storing instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device implements the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having computer program instructions stored thereon; characterized in that: When the computer program instructions are executed by an electronic device, the electronic device implements the method according to any one of claims 1 to 8.

Images