Photographing method, electronic equipment and storage medium

Abstract

The invention provides a photographing method, electronic equipment and a storage medium, relates to the field of terminal equipment, and can solve the problem that stripes with different brightness degrees exist on a snapshot image and improve the image quality of the snapshot image. The method comprises the following steps: starting a camera application in the electronic equipment; in response to a snapshot mode setting operation input by a user in the camera application, starting a snapshot function, and detecting a banding phenomenon; in the presence of banding, acquiring images based on a first sensor mode, and generating a first image queue and a second image queue; determining a wonderful moment, and selecting a target image frame based on the wonderful moment; and in response to a photographing instruction, taking a third image in the target image frame as a reference frame, fusing the reference frame with a fourth image in the target image frame, and generating a snapshot image.

Description

Technical Field

[0001] The application relates to the technical field of terminals, and particularly relates to a photographing method, an electronic device, and a storage medium. Background Art

[0002] With the continuous development of electronic device technology, more and more electronic devices support multiple photographing modes. When a user needs to capture a wonderful moment through an electronic device, the user can set the camera application in the electronic device to work in the capture mode so as to capture a wonderful image of the subject. However, in the capture mode, if the brightness changes during the process of the image sensor in the electronic device collecting an image, the exposure of different rows in the image frame will be different, resulting in stripes with different brightness levels on the captured image generated by the electronic device and affecting the user's photographing experience. Summary of the Invention

[0003] Embodiments of the present application provide a photographing method, an electronic device, and a storage medium, which can solve the problem of stripes with different brightness levels on the captured image and improve the image quality of the captured image.

[0004] To achieve the above object, the embodiments of the present application adopt the following technical solutions:

[0005] In a first aspect, an embodiment of the present application provides a photographing method applied to an electronic device. The method includes: starting a camera application in the electronic device; in response to a capture mode setting operation input by a user in the camera application, turning on a capture function and detecting a banding phenomenon; in the case of the existence of banding, collecting an image based on a first sensor mode and generating a first image queue and a second image queue; at least one first image in the first image queue is an image collected by the electronic device under a first exposure duration, and at least one second image in the second image queue is an image collected by the electronic device under a second exposure duration, and the first exposure duration is greater than the second exposure duration; determining a wonderful moment and selecting a target image frame based on the wonderful moment; in response to a photographing instruction, using a third image in the target image frame as a reference frame and fusing it with a fourth image in the target image frame to generate a captured image; where the exposure duration of the third image is the second exposure duration, and the fourth image includes images in the target image frame other than the third image.

[0006] Based on this solution, in the capture scenario, it is detected whether there is banding. In the case of the existence of banding, the dual-exposure mode is used to collect images. When the capture instruction is triggered, the short frame is used as a reference frame to be fused with the long frame to generate a captured image. Since the exposure duration of the short frame is short, the brightness and darkness effect of the short frame is good, and there is almost no banding phenomenon. Therefore, using the short frame as a reference frame to be fused with the long frame can obtain a captured image without the banding phenomenon, and the quality of the captured image is high.

[0007] In a possible implementation manner of the first aspect, the above-mentioned detection of the banding phenomenon includes: obtaining the frequency of the light source where the electronic device is located; determining whether there is banding according to the frequency of the light source.

[0008] Based on this solution, by obtaining the frequency of the light source in the environment where the electronic device is located and determining whether there is banding according to the light source frequency, it is possible to accurately determine whether there is banding in the preview stage.

[0009] In a possible implementation manner of the first aspect, the above-mentioned determination of whether there is banding according to the frequency of the light source includes: determining that there is banding when the frequency of the light source is greater than or equal to the frequency threshold; determining that there is no banding when the frequency of the light source is less than the frequency threshold.

[0010] Based on this solution, when the light source frequency is large, the duration of one strobing of the light source is small. Then, when collecting images in the single-exposure mode, the brightness during exposure will change due to the strobing of the light source, resulting in banding in the collected images. When the light source frequency is small, the duration of one strobing of the light source is long. Then, when collecting images in the single-exposure mode, the strobing of the light source may not cause the brightness during exposure to change. Therefore, there is no banding. That is to say, by comparing the light source frequency with the frequency threshold, it is possible to relatively accurately determine whether there is banding. Exemplarily, the frequency threshold can be determined based on the exposure duration corresponding to the single-exposure mode.

[0011] In a possible implementation manner of the first aspect, the above method further includes: generating a preview image based on the first image in the case of the existence of banding; displaying the preview image on the preview interface.

[0012] Based on this solution, in the preview stage of the capture scenario of this application, by sending the long frame for display and not sending the short frame for display, it can ensure that the quality of the preview image seen by the user is good and improve the preview effect. Moreover, since the screen refresh rate is relatively fast in the preview stage, when sending the long frame for display in the preview stage, the user will not see the banding caused by the change of the light source brightness.

[0013] In a possible implementation of the first aspect, the above-mentioned selection of the target image frame based on the wonderful moment includes: determining the sensor mode corresponding to the wonderful moment based on the wonderful moment; the sensor mode includes a first sensor mode and a second sensor mode; in the case where the sensor mode is the first sensor mode, select the target image frame from the first image queue and the second image queue.

[0014] Based on this solution, after determining the wonderful moment, it is possible to determine whether the double-exposure mode or the single-exposure mode is adopted for the wonderful moment. If the double-exposure mode is adopted for the wonderful moment, then frame selection can be performed in the long-frame queue and the short-frame queue to ensure that the selected frame can generate the captured image that the user expects to shoot.

[0015] In a possible implementation of the first aspect, the above-mentioned enabling of the capture function in response to the capture mode setting operation input by the user in the camera application includes: responding to the first operation input by the user to the automatic capture control in the preview interface, and enabling the automatic capture function; or, responding to the second operation input by the user to the manual capture control in the preview interface, and enabling the manual capture function; where the capture mode setting operation includes the first operation and the second operation.

[0016] Based on this solution, the manual capture function can be enabled in the preview interface, or the automatic capture function can be enabled in the preview interface. In both the manual capture and automatic capture scenarios, the short frame can be used as a reference frame to be fused with the long frame to ensure that the generated captured image has no banding and the quality of the captured image is good.

[0017] In a possible implementation of the first aspect, in the case where the manual capture function is enabled, the capture instruction is the capture instruction input by the user in the preview interface; in the case where the automatic capture function is enabled, the capture instruction is the instruction generated by the electronic device when detecting the wonderful moment.

[0018] Based on this solution, in the manual capture scenario, the user needs to input a capture operation in the preview interface to trigger the generation of the capture instruction to generate the captured image; in the automatic capture scenario, there is no need for the user to input a capture operation in the preview interface, and the sensing module in the electronic device can automatically trigger the capture instruction when detecting the wonderful moment.

[0019] In a possible implementation of the first aspect, the above method further includes: responding to the virtualization enabling operation input by the user in the preview interface, and enabling the virtualization function; in the case where the virtualization function is enabled, generating a virtualized captured image based on the wonderful moment and the captured image.

[0020] Based on this solution, in the scenario of capturing a photo, the user can enable the defocus function. When the user enables the defocus function, after generating the captured photo, it is necessary to further perform defocus processing on the captured photo to obtain a defocused captured photo.

[0021] In a possible implementation manner of the first aspect, generating a defocused captured photo based on the wonderful moment and the captured photo includes: determining a fifth photo corresponding to the wonderful moment in the auxiliary road photo queue based on the wonderful moment; wherein, the photos in the first photo queue and the second photo queue are the photos captured by the first camera of the electronic device, and the photos in the auxiliary road photo queue are the photos captured by the second camera of the electronic device, and the first camera is different from the second camera; determining the parallax based on the fifth photo and the captured photo; performing defocus processing on the captured photo based on the parallax to obtain a defocused captured photo.

[0022] Based on this solution, the captured photo can be used as the main road photo, and then the auxiliary road photo (i.e., the fifth photo) can be obtained from the auxiliary road queue (also referred to as the auxiliary road photo queue) based on the wonderful moment. Since the main road photo and the auxiliary road photo are the photos captured by different cameras, there is a parallax between the main road photo and the auxiliary road photo, and the captured photo can be defocused according to this parallax.

[0023] In a possible implementation manner of the first aspect, the above method further includes: in the case of no banding, capturing photos based on the second sensor mode and generating a third photo queue; at least one frame of the photos in the third photo queue is the photo captured by the electronic device under the first exposure duration.

[0024] Based on this solution, in the case of no banding, single-exposure mode is used to capture photos to obtain a photo frame queue. Since there is no banding when using single-exposure mode to capture photos, it is only necessary to continue using single-exposure mode to capture photos without switching the sensor mode. If banding is detected during the preview stage when using single-exposure mode to capture photos, the sensor mode is switched to double-exposure mode to ensure that the generated captured photo has no banding.

[0025] In the second aspect, an embodiment of the present application further provides a photographing device, which can be applied to an electronic device. The functions of the device can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions to implement any one of the above photographing methods.

[0026] In a third aspect, the present application provides an electronic device, which includes a display screen, a memory, and one or more processors; the display screen and the memory are coupled to the processor; wherein, computer program code is stored in the memory, and the computer program code includes computer instructions. When the computer instructions are executed by the processor, the electronic device is caused to execute the photographing method provided in the first aspect and any possible design thereof.

[0027] In a fourth aspect, the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are run on an electronic device, the electronic device is caused to execute the photographing method provided in the first aspect and any possible design thereof.

[0028] In a fifth aspect, the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device is caused to execute the photographing method provided in the first aspect and any possible design thereof.

[0029] It can be understood that for the beneficial effects that can be achieved by the technical solutions provided in the second to fifth aspects above, reference may be made to the beneficial effects in the first aspect and any possible design thereof, which will not be elaborated herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of a captured image with banding phenomenon provided by an embodiment of the present application;

[0031] Figure 2 Schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application;

[0032] Figure 3 Schematic diagram of the software architecture of an electronic device provided by an embodiment of the present application;

[0033] Figure 4 Schematic diagram of the software architecture of another electronic device provided by an embodiment of the present application;

[0034] Figure 5 Schematic diagram of the flowchart of a photographing method provided by an embodiment of the present application;

[0035] Figure 6 Schematic diagram of starting a camera application provided by an embodiment of the present application;

[0036] Figure 7 Schematic diagram of setting a capture mode provided by an embodiment of the present application;

[0037] Figure 8 Schematic diagram of enabling an automatic capture function or a manual capture function provided by an embodiment of the present application;

[0038] Figure 9 Another schematic diagram for enabling the automatic capture function provided by the embodiment of the present application;

[0039] Figure 10 A schematic diagram for enabling the defocusing function provided by the embodiment of the present application;

[0040] Figure 11 A schematic diagram for frame selection based on wonderful moments provided by the embodiment of the present application;

[0041] Figure 12 A schematic diagram for triggering the photographing instruction on the preview interface provided by the embodiment of the present application;

[0042] Figure 13 A schematic diagram for blurring the captured image provided by the embodiment of the present application;

[0043] Figure 14 A schematic flowchart of another photographing method provided by the embodiment of the present application. Detailed implementation manners

[0044] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular forms "a", "an", "the", "above", "said", "this" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that " / " means "or", for example, A / B may mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone. These three situations.

[0045] Referring to "embodiment" in the present application means that the specific features, structures or characteristics described in combination with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art understand explicitly and implicitly that the embodiments described in the present application can be combined with other embodiments.

[0046] The terms "first" and "second" in the following embodiments of the present application are only for descriptive purposes, and cannot be construed as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, unless otherwise stated, the meaning of "a plurality" is two or more.

[0047] First, the nouns involved in the embodiments of the present application are described as follows:

[0048] Banding phenomenon: When an image sensor captures an image by line-by-line exposure, if the energy received by pixel points in different lines is different, it will cause bright and dark stripe phenomena on the captured image generated by the electronic device, that is, the Banding phenomenon. For example, when an image sensor captures an image, if the brightness changes during exposure due to reasons such as light source strobing, the exposure of different lines in a frame of image will be different, resulting in bright and dark changing stripes on the captured image.

[0049] Single-exposure mode: It means that the image sensor only exposes the object to be photographed once when capturing an image, that is, when the image sensor operates in the single-exposure mode, only one frame of image is captured.

[0050] Dual-exposure mode: It means that the image sensor can expose the object to be photographed twice when capturing an image, that is, when the image sensor operates in the dual-exposure mode, two frames of images can be captured at once, and the exposure durations of the two frames of images are different. For example, in the dual-exposure mode, the object to be photographed is exposed twice based on the first exposure duration and the second exposure duration. The two frames of images captured by the image sensor are a long frame and a short frame respectively, and the exposure duration of the long frame (such as the first exposure duration) is longer than that of the short frame (such as the second exposure duration).

[0051] Highlight moment: It refers to the moment when the state and / or action of the object to be photographed are in the best condition within a period of time. In some examples, the object to be photographed can be in a static state or in a moving state. When the object to be photographed is in a static state, the highlight moment is the moment when the object to be photographed is clearly imaged and without jitter. For example, when taking a portrait close-up, the moment when the person opens their eyes and smiles and is clearly imaged is the highlight moment. When the object to be photographed is in a moving state, the highlight moment is the image of the moment when the object to be photographed is in motion. For example, when taking a picture of a person jumping, the moment when the person is in the air after taking off and is clearly imaged is the highlight moment.

[0052] Motion Capture algorithm for action capture: It refers to a technology that captures the motion information of the object to be photographed and converts it into a data format recognizable by a computer for processing and analysis. When using Motion Capture to process an image, one or more frames can be captured first as reference frames. The reference frames are used to provide information such as the initial position and pose of the object to be photographed as a reference for subsequent frames; then, according to the information of the reference frames, subsequent frames are corrected and adjusted to more accurately capture the motion trajectory of the object to be photographed.

[0053] In an embodiment of the present application, when the image sensor acquires an image in a dual-exposure mode, the MotionCapture algorithm can fuse the short frame as a reference frame with the long frame. Since the exposure duration of the short frame is short and the light and dark effect is good, there is almost no banding problem. The exposure duration of the long frame is long and the image quality is good. Therefore, when the short frame is used as a reference frame to fuse with the long frame with good image quality, it can not only ensure that the captured image generated has no banding problem, but also ensure that the captured image has good image quality.

[0054] Generally, in a capture scenario, the image sensor in an electronic device can perform a single exposure on the subject to obtain a captured image. When the image sensor acquires an image in a single-exposure mode, if the exposure duration is greater than the stroboscopic duration of the light source, then during the exposure of the subject, the exposure of different rows in the captured image frame will be different due to the stroboscopic effect of the light source, resulting in bright and dark changing stripes in the captured image captured by the electronic device, which affects the user's shooting experience.

[0055] Figure 1 It is a schematic diagram of a captured image with a banding phenomenon provided by an embodiment of the present application. As Figure 1 shown, when capturing the jumping action of the subject, due to the change in brightness during exposure, there are bright and dark changing stripes in the captured image captured.

[0056] To solve the above problems, an embodiment of the present application provides a photographing method. In a capture scenario, by detecting whether there is banding, in the case of banding, the image sensor is driven to acquire an image in a dual-exposure mode, and when a photographing instruction is triggered, the short frame is used as a reference frame to fuse with the long frame to generate a captured image. Since the exposure duration of the short frame is short, the light and dark effect of the short frame is good and there is almost no banding phenomenon. Therefore, fusing the short frame as a reference frame with the long frame can obtain a captured image without banding phenomenon and with high quality.

[0057] Moreover, when acquiring an image in a dual-exposure mode, in order to improve the preview effect, by sending the long frame for display and not sending the short frame for display during the preview stage, it can ensure that the quality of the preview image seen by the user is good. And since the screen refresh rate is relatively fast during the preview stage, the user will not see the banding phenomenon when sending the long frame for display during the preview stage.

[0058] The technical solution provided by the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0059] The technical solution provided by this application can be applied to an electronic device with a capture function. In some embodiments, the electronic device may be a mobile phone, a tablet computer, a handheld computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, and / or a smart city device, etc. The specific type of the electronic device is not particularly limited in the embodiments of this application.

[0060] Exemplarily, taking the electronic device as a mobile phone as an example, Figure 2 The structural schematic diagram of an electronic device provided by the embodiments of this application is shown.

[0061] Referring to Figure 2 As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a display screen 193, a subscriber identification module (SIM) card interface 194, and a camera 195, etc. Among them, the sensor module 180 may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0062] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0063] The controller may be the nerve center and command center of the electronic device. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0064] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0065] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0066] The charging management module 140 is used to receive a charging input from a power supply device (such as a charger, laptop power supply, etc.). Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input of the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device.

[0067] While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141. Among them, the battery 142 can specifically be composed of multiple batteries connected in series. The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110.

[0068] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives the input from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the display screen 193, the camera 195, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the voltage, current, number of battery cycles, and battery health status (leakage, impedance) of the battery. In some other embodiments, the power management module 141 can also be disposed in the processor 110.

[0069] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external non-volatile memory.

[0070] The internal memory 121 can include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 110 and can be used to store the operating system or the executable programs of other running programs (such as machine instructions), and can also be used to store the data of users and application programs, etc. The non-volatile memory can also store executable programs and store the data of users and application programs, etc., and can be pre-loaded into the random access memory for the processor 110 to directly read and write.

[0071] A touch sensor, also known as a "touch control device". The touch sensor can be disposed on the display screen 193, and together with the display screen 193, they form a touch screen, also known as a "touch control screen". The touch sensor is used to monitor touch operations applied thereto or in its vicinity. The touch sensor can transmit the monitored touch operations to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 193. In some other embodiments, the touch sensor can also be disposed on the surface of the electronic device, at a different position from that of the display screen 193.

[0072] The ambient light sensor is used to sense the ambient light brightness. For example, the ambient light sensor can measure the light intensities of four channels of the ambient light. The ambient light sensor outputs the measured light intensities of the four channels of the ambient light to the processor 110. The processor 110 can process the light intensities of the four channels of the ambient light output by the ambient light sensor to obtain the light intensity of the ambient light. In the screen-on state, the electronic device can adaptively adjust the display screen brightness according to the obtained light intensity of the ambient light.

[0073] The pressure sensor is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor can be disposed on the display screen 193. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. When a touch operation is applied to the display screen 193, the electronic device monitors the intensity of the touch operation according to the pressure sensor. The electronic device can also calculate the position of the touch according to the monitoring signal of the pressure sensor. In some embodiments, touch operations with the same touch position but different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than the first pressure threshold is applied to the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold is applied to the short message application icon, the instruction to create a new short message is executed.

[0074] In some embodiments, the electronic device can include one or N cameras 195, where N is a positive integer greater than 1. In the embodiments of the present application, the types of the cameras 195 can be distinguished according to the hardware configuration and the physical position. For example, the camera disposed on the side of the display screen 193 of the electronic device can be called a front camera, and the camera disposed on the back cover of the electronic device can be called a rear camera; for another example, a camera with a short focal length and a large viewing angle can be called a wide-angle camera, and a camera with a long focal length and a small viewing angle can be called a normal camera. Among them, the length of the focal length and the size of the viewing angle are relative concepts and there are no specific parameter limitations. Therefore, the wide-angle camera and the normal camera are also relative concepts, and can be specifically distinguished according to physical parameters such as the focal length and the viewing angle.

[0075] The electronic device realizes the display function through the GPU, the display screen 193, the application processor, etc. The GPU is a microprocessor for image editing, connecting the display screen 193 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0076] The electronic device can realize the shooting function through the ISP, the camera 195, the video codec, the GPU, the display screen 193, the application processor, etc. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information. In the embodiments of the present application, during the frame drawing process of each image frame, the functions of the GPU are used to make the finally displayed picture obtain better display effects and performance.

[0077] The ISP is used to process the data fed back by the camera 195. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera photosensitive element. The optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise and brightness of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP may be set in the camera 195. The camera 195 is used to capture static images or videos.

[0078] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0079] The display screen 193 is used to display images, videos, etc. The display screen 193 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 193, where N is a positive integer greater than 1.

[0080] In the embodiments of the present application, the display screen 193 can be used to display pages required by the electronic device (for example, wizard pages (including highlight recommendation pages and external module access pages), etc.), and display images captured by any one or more cameras 195 in this interface.

[0081] The wireless communication function of the electronic device can be implemented by antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem, baseband processor, etc.

[0082] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas.

[0083] The mobile communication module 150 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device. The mobile communication module 150 can receive electromagnetic waves through antenna 1, and perform filtering, amplification and other processing on the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through antenna 1 and radiate it out. In some embodiments, at least some functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be disposed in the same device.

[0084] The modulation and demodulation processor can include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to speaker 170A, receiver 170B, etc.), or displays an image or video through the display screen 193. In some embodiments, the modulation and demodulation processor can be an independent device. In other embodiments, the modulation and demodulation processor can be independent of the processor 110 and be disposed in the same device as the mobile communication module 150 or other functional modules.

[0085] The wireless communication module 160 may provide solutions for wireless communications applied to an electronic device, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSSs), frequency modulation (FM), near field communication (NFC), infrared (IR), and the like. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, frequency-modulate them, amplify them, and convert them into electromagnetic waves via the antenna 2 for radiation.

[0086] The SIM card interface 194 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 194 to achieve contact and separation from the electronic device. The electronic device may support one or more SIM card interfaces. The SIM card interface 194 may support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards may be inserted into the same SIM card interface 194 at the same time. The SIM card interface 194 may also be compatible with external memory cards. The electronic device interacts with the network through the SIM card to implement functions such as calls and data communication. One SIM card corresponds to one user number.

[0087] It can be understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are only illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods or combinations of multiple interface connection methods in the above embodiments.

[0088] Of course, it can be understood that the above Figure 2 is only an exemplary illustration when the form of the electronic device is a mobile phone. When the electronic device is in other form such as a tablet computer, a handheld computer, a PC, a PDA, a wearable device (such as a smart watch, a smart bracelet), etc., the structure of the electronic device may include fewer structures than those Figure 2 shown, or may include more structures than those Figure 2 shown, which is not limited herein.

[0089] It can be understood that, generally speaking, in addition to the support of hardware, the realization of the functions of an electronic device also requires the cooperation of software. The software system of an electronic device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of the present application, taking the system as an example, the software structure of the electronic device will be exemplarily described.

[0090] Figure 3 FIG. is a schematic diagram of the layered architecture of the software system of the electronic device provided in the embodiments of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces (such as APIs).

[0091] In some examples, as shown in Figure 3 , in the embodiments of the present application, the software of the electronic device is divided into five layers, from top

[0092] to bottom, they are the application layer, the application framework layer, the system library and runtime ( runtime), the hardware abstraction layer (hardware abstraction layer, HAL), and the driver layer (or called the kernel layer).

[0093] Among them, the application layer may include a series of applications. As Figure 3 shown, the application layer may include applications such as cameras, galleries, calendars, maps, WLAN, Bluetooth, music, videos, text messages, calls, etc.

[0094] The framework layer provides application programming interfaces (application programming interfaces, APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions or services. For example, the application framework layer may include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, etc., and the embodiments of the present application do not make any restrictions on this.

[0095] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0096] The content provider is used to store and obtain data, and make this data accessible to applications. This data may include videos, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc.

[0097] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying pictures. In some embodiments, the view system can also include or start a rendering thread to complete operations such as drawing the frame buffer.

[0098] The telephone manager is used to provide the communication function of the electronic device. For example, the telephone manager can manage the call status of the call application (including initiation, connection, hanging up, etc.).

[0099] The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

[0100] The notification manager enables the application to display notification information in the status bar. It can be used to convey notification-type messages, which can disappear automatically after a short stay without user interaction. For example, the notification manager is used to inform that the download is complete, message reminder, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as the notification of a background-running application, or a notification that appears on the screen in the form of a dialogue window. For example, it prompts text information in the status bar, emits a prompt tone, the electronic device vibrates, the indicator light flashes, etc.

[0101] The system library can include multiple functional modules. For example: surface manager, Media Libraries, 3D graphics processing library (e.g., OpenGL ES), 2D graphics module (e.g., SGL), etc. The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The Media Libraries support the playback and recording of multiple common audio and video formats, as well as static image files, etc. The Media Libraries can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. SGL is a drawing module for 2D drawing.

[0102] The Android Runtime includes core libraries and a virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part is the functional functions that the Java language needs to call, and the other part is the core libraries of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0103] The HAL layer is an interface layer located between the operating system kernel and the hardware circuit. Its purpose is to abstract the hardware. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable across multiple platforms. The HAL layer provides a standard interface to display the device hardware functions to a higher-level Java API framework (i.e., the framework layer). The HAL layer contains multiple library modules, and each module implements an interface for a specific type of hardware component. For example: the audio HAL audio module, the bluetooth HAL bluetooth module, the camera HAL camera module (which can also be called the camera HAL or the camera hardware abstraction module), and the sensors HAL sensor module (or called the Isensor service, sensor service).

[0104] The kernel layer is the layer between the hardware and the software. The kernel layer includes at least a display driver, a camera driver, an audio driver, a sensor driver, a battery driver, etc., which are not limited in this application. Among them, the sensor driver can specifically include the drivers for each sensor included in the electronic device, such as an ambient light sensor driver. Exemplarily, the ambient light sensor driver can send the detection data of the ambient light sensor to the sensing module in a timely manner in response to an indication or instruction from the sensor module to obtain detection data.

[0105] In the embodiments of this application, as Figure 4 shown, the application layer includes a camera application. The application framework layer may include a camera access interface, and the camera access interface is used to provide an application programming interface and a programming framework for the camera application. The HAL layer is an interface layer between the application framework layer and the driver layer ( Figure 4 not shown in the figure) and can provide a virtual hardware platform for the operating system.

[0106] As Figure 4As shown, in the embodiment of the present application, after detecting the start operation of the user to open the camera application, the camera application calls the camera access interface of the framework layer to start the camera application, and then drives the hardware devices (such as image sensors and detection devices) to start by calling the camera module in the HAL layer. For example, the camera module in the HAL layer can send a start instruction to the camera device driver in the driver layer, and the camera device driver can drive the image sensor corresponding to the camera device to collect images and then send the collected images back to the HAL layer. After the camera of the electronic device is started, the detection device can detect the frequency of the light source where the electronic device is located and send the detected light source frequency back to the 3A module in the HAL layer, and the 3A module detects whether there is banding according to the light source frequency.

[0107] When the 3A module determines that there is banding, the 3A module can send indication information to the camera module to instruct the camera module to drive the image sensor corresponding to the camera device to collect images in the dual-exposure mode. When the image sensor collects images in the dual-exposure mode, the image front end (IFE) module can fuse the long frame and the short frame and then send it for display, or can send the long frame for display and not send the short frame for display. Since the exposure time of the long frame is longer than that of the short frame, the image quality of the long frame is higher than that of the short frame. In order to ensure better image quality in the preview stage, when there is banding, the AE module can output a flag bit to the IFE module to instruct the IFE module not to fuse the long frame and the short frame and directly send the long frame for display and not send the short frame for display to ensure better quality of the preview image. When there is no banding, the image sensor still collects images in the single-exposure mode.

[0108] In the preview stage, the perception module of the HAL layer can detect wonderful images in real time. When the perception module detects a wonderful image, it can determine the moment corresponding to the wonderful image as the wonderful moment. The perception module can send the time stamp corresponding to the wonderful moment to the decision module, so that the decision module can determine whether the double-exposure mode or the single-exposure mode is adopted at the wonderful moment according to the time stamp. When the decision module determines that the sensor mode corresponding to the wonderful moment is the double-exposure mode, the frame selection module can select target image frames from the long frame queue (also called the first image queue) and the short frame queue (also called the second image queue) based on the wonderful moment; when the decision module determines that the sensor mode corresponding to the wonderful moment is the single-exposure mode, the frame selection module can select target image frames from an image frame queue based on the wonderful moment. When the electronic device automatically triggers a photographing instruction or the user manually triggers a photographing instruction, the frame selection module sends the selected target image frames to the photographing algorithm module. When the sensor mode corresponding to the wonderful moment is the double-exposure mode, the target image frames include a long frame and a short frame, and the photographing algorithm module can fuse the short frame with the long frame based on the target image frames to ensure that there is no banding problem in the generated captured image.

[0109] In some examples, the user can turn on the defocus function in the captured scene. As Figure 4 shown, the HAL layer can also include a defocus module. When the user turns on the defocus function (such as background defocus), the defocus module can perform defocus processing on the captured image generated by the photographing algorithm module to obtain a defocused image.

[0110] The technical solutions provided in the embodiments of the present application can all be implemented in an electronic device having the above-mentioned hardware architecture or software architecture.

[0111] Based on the above Figure 3 and Figure 4 shown software architecture, the following will introduce the photographing method provided in the embodiments of the present application in combination with Figure 5 shown. Figure 5 is a schematic flowchart of a photographing method provided in an embodiment of the present application. As Figure 5 shown, the photographing method can include the following steps S501-S515.

[0112] S501. In response to a start instruction input by the user to the camera application of the electronic device, start the camera application.

[0113] Exemplarily, the start instruction can be a start operation or a voice start instruction, and the embodiments of the present application do not limit the manner of starting the camera application. The following embodiments will be described by taking the start instruction as a start operation as an example.

[0114] For example, taking the start instruction as a start operation as an example, as Figure 6As shown in (a) therein, the electronic device displays a desktop, and the desktop includes an icon of the camera application. The user can click on the icon of the camera application on the desktop. The electronic device receives and responds to the click operation (i.e., the startup operation) on the icon of the camera application, and the electronic device starts the camera application and displays Figure 6 the preview interface 601 shown in (b) therein. That is, the above startup instruction can also be considered as an instruction for starting the image preview function of the camera application. It should be noted that the photographing method in the embodiments of the present application can be applied to any application program that supports the capture function. For example, the above camera application can be the system camera application or a third-party application with the capture function.

[0115] Exemplarily, after the user starts the camera application, the camera application can send a preview request request to the camera access interface in the framework layer. The camera access interface sends the preview request to the camera module in the HAL layer, and the camera module sends the preview request to the camera device driver in the driver layer to drive the camera to start. After the camera starts, the image sensor can collect the image optical signal, and transmit the image optical signal to the image signal processor for preprocessing to obtain a raw image, and then transmit the raw image back to the HAL layer through the camera device driver. The raw image is the image obtained after the photosensitive element in the camera converts the captured light source signal into an electrical signal, and can also be called the original image.

[0116] In some examples, after the camera application starts, it can continuously generate multiple preview requests and sequentially send each preview request to the function modules related to images in the electronic device. The function modules related to images in the electronic device respond to each preview request, collect the raw image and generate the preview image corresponding to each preview request based on the raw image.

[0117] S502. In response to the capture mode setting operation input by the user, display the preview interface corresponding to the capture mode.

[0118] Exemplarily, the camera application includes multiple shooting modes, such as portrait mode, night scene mode, and capture mode, etc. In the portrait mode, the facial features of the person in the captured image are obvious. In the capture mode, the actions of the shooting object at the moment of movement can be captured in the captured image. In the night scene mode, the clarity of the captured image is relatively high. The photographing method provided in the embodiments of the present application can be applied in the capture mode. The following embodiments will exemplarily illustrate the photographing method provided in the embodiments of the present application by taking the capture mode as an example.

[0119] In some embodiments, the user can input one operation or multiple operations in the electronic device to turn on the capture function. The one operation or multiple operations can both be called the capture mode setting operation.

[0120] For example, as Figure 7As shown in (a) of [], the menu bar below the preview interface 701 includes a capture mode control 71. The user can click on the capture mode control 71 in the preview interface 701 (this click operation is the capture mode setting operation). In response to the click operation input by the user on the capture mode control 71, the capture function is enabled and the Figure 7 preview interface 702 corresponding to the capture mode shown in (b) of [] is displayed. In some examples, the user can also input a swipe operation (i.e., the capture mode setting operation) in the Figure 7 preview interface 701 shown in (a) of [] to slide the focus position to the capture mode control 71, thereby enabling the capture function and displaying the Figure 7 preview interface 702 corresponding to the capture mode shown in (b) of [].

[0121] The capture mode includes an automatic capture mode and a manual capture mode. In the automatic capture mode (i.e., when the automatic capture function is enabled), the electronic device can automatically identify exciting moments and capture images without the user performing a capture operation, which not only saves user operations but also avoids missing exciting moments. In the manual capture mode (i.e., when the manual capture function is enabled), the electronic device can identify exciting moments during the preview stage and select frames according to the exciting moments. When the user triggers a capture instruction (such as clicking on a capture control) in the preview interface, the capture algorithm module can fuse the image frames selected during the preview stage to generate a captured image. That is to say, in the automatic capture mode, a captured image can be generated without the user triggering a capture instruction, while in the manual capture mode, a user needs to trigger a capture instruction to generate a captured image.

[0122] To avoid the problem that in the automatic capture mode, too many captured images are generated, occupying too much memory space and affecting the use of the electronic device, and too many captured images of the same shooting scene cause redundancy and affect the user experience. In the automatic capture mode, a capture quantity threshold can be preset in advance. After the electronic device identifies a capture scene, no more than the capture quantity threshold of captured images are generated in each capture scene. For example, if the number of captured images of the current capture scene reaches the quantity threshold, the electronic device will no longer perform automatic capture on this capture scene.

[0123] The following combines Figure 8 and Figure 9 to give an exemplary description of the ways for the user to set the automatic capture function and the manual capture function.

[0124] For example, as Figure 8 shown in (a) of [], the preview interface 801 includes a manual capture control 81. The user can click on the manual capture control 81 in the preview interface 801 (this click operation is the capture mode setting operation, also referred to as the second operation). In response to the click operation of the user on the manual capture control 81, the manual capture function is enabled and theFigure 8 The preview interface 802 shown in (b) of [[reference]]. It should be noted that the color of the manual capture control 81 can be different when the manual capture function is turned on and off, so that the user can see whether the manual capture function is enabled in the preview interface. For example, when the manual capture function is turned off, the manual capture control 81 is not filled with color, and when the manual capture function is turned on, the manual capture control 81 is filled with black or yellow. When the user needs to turn off the manual capture function, they can click on the manual capture control 81 again in the preview interface 802 to turn off the manual capture function.

[0125] For another example, as Figure 8 shown in (c) of [[reference]], the preview interface 803 may further include an automatic capture control 82. The user can click on the automatic capture control 82 in the preview interface 803 (this click operation is the capture mode setting operation, also referred to as the first operation). In response to the user's click operation on the automatic capture control 82, the automatic capture function is enabled.

[0126] In the embodiments of the present application, there is no limitation on the setting position of the manual capture control 81 and the automatic capture control 82 in the preview interface. The automatic capture control 82 can be set at Figure 8 the lower right position of the preview interface 803 shown in (c) of [[reference]], or can be set at Figure 8 the upper position of the preview interface 804 shown in (d) of [[reference]]. As Figure 8 shown in (d) of [[reference]], the user can click on the automatic capture control 82 in the preview interface 804. In response to the user's click operation on the automatic capture control 82, the automatic capture function is enabled. When the user needs to turn off the automatic capture function, they can click on the automatic capture control 82 again in the preview interface 804 to turn off the automatic capture function.

[0127] In some embodiments, the preview interface of the electronic device may not include an automatic capture control. In this case, the automatic capture function can be enabled through the setting control in the preview interface.

[0128] For example, as Figure 9 shown in (a) of [[reference]], to enter the capture mode, the user can click on the setting control 91 in the preview interface 901. In response to the user's click operation on the setting control 91, a setting interface as shown in Figure 9 (b) of [[reference]] is displayed. As Figure 9 shown in (b) of [[reference]], the setting interface includes multiple controls related to taking pictures (such as photo ratio control, intelligent photo control, and filter photo control) and multiple controls related to videos (such as video resolution control, video frame rate control, high-efficiency video format control, and one-shot multi-capture control). The user can perform a click operation on the intelligent photo control 92 in the Figure 9 setting interface shown in (b) of [[reference]]. In response to the user clicking on the intelligent photo control 92,Figure 9 The intelligent photo-taking setting interface shown in (c) therein. The intelligent photo-taking setting interface includes a voice-controlled photo-taking control, a gesture photo-taking control, a smiling face capture control, and an automatic capture control. The electronic device defaults to turning off the automatic capture function, that is, the slider in the automatic capture control 93 shown in (c) as Figure 9 shown is on the left side. The user can Figure 9 perform a click or slide operation on the automatic capture control 93 in the intelligent photo-taking setting interface shown in (c) therein. In response to the user clicking or sliding the automatic capture control 93, as Figure 9 shown in (d) therein, the electronic device can move the slider in the automatic capture control 93 to the right side and turn on the automatic capture function. As Figure 9 shown in (d) therein, when the automatic capture function is turned on, the slider in the automatic capture control 93 moves to the right side, and the left area of the automatic capture control 93 is a predetermined color (such as blue).

[0129] In some embodiments, in the video recording mode (also referred to as the video shooting mode), the user can also turn on the automatic capture or manual capture function. When the automatic capture function is turned on in the video recording mode, in response to the user starting to record a video, the electronic device can detect wonderful moments in real time and select frames based on the wonderful moments to generate captured images. During the video recording process of the electronic device, in the playback control, the generated captured images can be updated in real time, and thumbnails of the captured images can be displayed in the playback control, so as to facilitate the user to understand the captured images taken. When the automatic capture function is turned on in the video recording mode, the processing flow of the following steps S503 - S514 can be performed during the video recording process (i.e., the recording stage); when the automatic capture or manual capture function is turned on in the photo-taking mode, steps S503 - S510 in the embodiments of the present application are executed in the preview stage, and steps S511 - S514 are executed in the photo-taking stage. The following embodiments illustrate the photo-taking method provided by the embodiments of the present application by taking automatic capture or manual capture in the photo-taking mode as an example.

[0130] In some examples, in order to highlight the subject, the user can turn on the blurring function in the capture mode. The embodiments of the present application do not limit the specific manner in which the user turns on the background blurring function. The user can turn on the blurring function through the blurring control or other means.

[0131] For example, taking the case where the user turns on the blurring function through the blurring control as an example, as Figure 10 shown, the user turns on the automatic capture function or the manual capture function ( Figure 10Taking the example of a user enabling the manual capture function for exemplary illustration), after that, the preview interface 1001 may include a blurring control 101. The user can click on the blurring control 101 in the preview interface 1001 (which can also be referred to as a blurring activation operation). In response to the user's click operation on the blurring control 101, the blurring function is enabled. In some examples, the color of the blurring control 101 is different when the blurring function is enabled and disabled.

[0132] It should be noted that in the automatic capture scenario or the manual capture scenario, when the user enables the blurring function, after the photographing algorithm module generates the captured image, the blurring module further needs to perform blurring processing on the captured image to obtain a blurred image.

[0133] S503. During the preview stage, the detection device continuously detects the light source information and sends the light source information to the 3A module.

[0134] Exemplarily, when previewing in the automatic capture or manual capture scenario, the detection device (such as a filter fliter) in the electronic device can continuously detect the information of the light source (such as the light source frequency) where the electronic device is located. For example, after the camera of the electronic device is started, the detection device in the electronic device can continuously detect the light source frequency so that the 3A module determines whether there is banding according to the light source frequency.

[0135] Exemplarily, 3A is a general term for auto exposure (AE), auto focus (AF), and auto white balance (AWB). The 3A module may include an AE module, an AF module, and an AWB module.

[0136] The AE module is used to automatically adjust the exposure parameters according to the ambient brightness of the electronic device to obtain a clear image. The AE module automatically adjusts the exposure time and aperture size by continuously monitoring the brightness of the scene, so that the camera can obtain an ideal exposure effect.

[0137] The AF module is used to automatically adjust the focusing distance of the lens to make the captured image clear. The AF module usually uses a motor-driven method, judges the focus position according to the algorithm, and automatically adjusts the position of the lens to achieve automatic focusing.

[0138] The AWB module is used to automatically adjust the color balance of the image to make the color of the captured image more realistic. The AWB module can automatically adjust the color parameters by detecting the color temperature of the scene, eliminate color difference and color cast phenomena, and make the color of the captured image more natural and more realistic.

[0139] S504. The 3A module performs filtering processing on the light source information detected by the detection device and detects whether there is banding.

[0140] Exemplarily, after the 3A module acquires the light source frequency through the detection device, it can filter the light source and compare the frequency of the filtered light source with the frequency threshold to determine whether there is a banding phenomenon. If the frequency of the filtered light source is greater than or equal to the frequency threshold, it is determined that there is banding; if the frequency of the filtered light source is less than the frequency threshold, it is determined that there is no banding.

[0141] The greater the frequency of the light source, the more stroboscopic times per unit time (e.g., per second). Then, when using the single-exposure mode for one exposure, the brightness is more likely to change, resulting in a banding phenomenon in the captured image. Therefore, when the frequency of the light source is greater than or equal to the frequency threshold, it is determined that there is banding. The smaller the frequency of the light source, the fewer stroboscopic times per unit time. Then, when using the single-exposure mode for one exposure, the brightness may not change. Therefore, when the frequency of the light source is less than the frequency threshold, it can be determined that there is no banding.

[0142] Exemplarily, the frequency threshold can be determined based on the exposure duration corresponding to the single-exposure mode. For example, when the frequency of the light source is greater than the frequency corresponding to the exposure duration of the single-exposure mode, it means that the duration of one stroboscopic flash of the light source is less than the exposure duration corresponding to the single-exposure mode. Then, when acquiring an image based on the single-exposure mode, the brightness during exposure will change due to the stroboscopic flash of the light source, resulting in different pixels in different rows receiving different energies and there being banding. For another example, when the frequency of the light source is less than the frequency corresponding to the exposure duration of the single-exposure mode, it means that the duration of one stroboscopic flash of the light source is greater than the exposure duration corresponding to the single-exposure mode. Then, when acquiring an image based on the single-exposure mode, the stroboscopic flash of the light source may not cause the brightness during exposure to change, so there will be no banding.

[0143] In some embodiments, the exposure duration corresponding to the single-exposure mode adopted by the image sensor can be the same as the first exposure duration (i.e., the exposure duration corresponding to the long frame) when the image sensor adopts the dual-exposure mode, and the above frequency threshold can also be determined based on the exposure duration corresponding to the long frame.

[0144] In the embodiments of the present application, when it is determined that there is banding, the image sensor can be driven to acquire images in the dual-exposure mode; when it is determined that there is no banding, the image sensor is driven to acquire images in the single-exposure mode. That is, the sensor mode when there is banding is different from the sensor mode when there is no banding. To avoid the frequent switching of the sensor mode in a short time from affecting the subsequent frame selection and image fusion of the photographing algorithm module, the 3A module can increase the frequency threshold or set the detection period when detecting whether there is banding to ensure that the sensor mode adopted by the image sensor does not change in a short time.

[0145] For example, when determining the frequency threshold based on the exposure duration corresponding to the single-exposure mode, a preset value can be added to the frequency corresponding to the exposure duration of the single-exposure mode to increase the frequency threshold and ensure that the image sensor does not frequently switch the sensor mode.

[0146] For another example, when 3A detects whether there is banding, it can be detected based on the detection period. That is, it is detected whether there is banding every preset detection period. Therefore, the sensor mode remains unchanged within each detection period, ensuring that the sensor mode does not frequently switch within a short period of time.

[0147] S505. In the case of banding, the AE module sends an indication message to the camera module and outputs a flag bit to the IFE module.

[0148] This indication message is used to instruct the camera module to drive the image sensor to collect images in the dual-exposure mode (such as the first sensor mode). That is, in the case of banding, the image sensor can collect images in the dual-exposure mode. For example, in the case of banding, the image sensor can expose the subject twice to obtain a long frame (such as the first image) corresponding to the first exposure duration and a short frame (such as the second image) corresponding to the second exposure duration, where the first exposure duration is greater than the second exposure duration.

[0149] Exemplarily, when the image sensor collects images in the dual-exposure mode, the image front-end IFE module can fuse (mergeraw) the collected long frame and short frame, or can not fuse the long frame (long raw) and short frame (short raw). Since the exposure durations of the long frame and short frame collected in the dual-exposure mode are quite different, if the long frame is used as the reference frame to be fused with the short frame in the preview stage, the effect of the generated fused image (such as mergeraw) will be poor. Since the exposure duration of the long frame is longer than that of the short frame, the image quality of the long frame is better. Therefore, in order to improve the preview effect in the preview stage of the capture scenario, the long frame can be sent for display and the short frame is not sent for display to ensure that the quality of the preview image seen by the user is better. It should be noted that since the screen refresh rate is relatively fast in the preview stage, when the long frame is sent for display in the preview stage, the user will not see the banding caused by the change in the light source brightness.

[0150] In some embodiments, in order to improve the preview effect, in the case of banding, the AE module can output a flag bit to the IFE module to instruct the IFE module not to fuse the long frame and short frame, but to send the long frame for display and not send the short frame for display.

[0151] In some embodiments, in the presence of banding, the image sensor can also collect images in a triple-exposure mode, that is, the image sensor can expose the object to be photographed three times to obtain a set of long frames, medium frames, and short frames (i.e., the image sensor outputs three frames at a time). Among them, the exposure duration of the long frame is longer than that of the medium frame, and the exposure duration of the medium frame is longer than that of the short frame. When the image sensor collects images in the triple-exposure mode, the preview stream includes three image frame queues, namely, the long frame queue, the medium frame queue, and the short frame queue, and the frame selection module supports selecting multiple groups of image frames from these three queues. In the embodiments of the present application, the sensor mode adopted by the image sensor in the presence of banding is not limited. For example, in the presence of banding, the image sensor can collect images in a double-exposure mode, a triple-exposure mode, or a more-exposure mode. Regardless of which sensor mode the image sensor adopts, a long frame can be sent for display and a short frame is not sent for display during the preview stage. In the embodiments of the present application, an example is given in which the image sensor collects images in a double-exposure mode when there is banding.

[0152] S506. The camera module drives the image sensor to collect long and short frames in a double-exposure mode based on the indication information.

[0153] Exemplarily, after receiving the indication information sent by the AE module, the camera module determines the presence of banding and sends a double-exposure notification to the camera driver in the driver layer according to the indication information. The camera driver drives the image sensor to collect images in a double-exposure mode according to the double-exposure notification. When the image sensor collects images in a double-exposure mode, the object to be photographed can be exposed twice at the same position. For example, the image sensor can expose the object to be photographed twice at the same position based on the first exposure duration and the second exposure duration respectively. The two exposures can obtain a first image (such as a long frame) with less noise but poorer light and dark effects, and a second image (such as a short frame) with more noise but better light and dark effects. It can be understood that when no banding is detected, the image sensor collects images in a single-exposure mode (such as the second sensor mode), and when banding is detected, the image sensor can be driven to switch the sensor mode from the single-exposure mode to the double-exposure mode. The exposure duration corresponding to the single-exposure mode can be the same as the first exposure duration, or different from the first exposure duration but longer than the second exposure duration.

[0154] When collecting images in a double-exposure mode, the electronic device can expose alternately according to the first exposure duration and the second exposure duration. The image frame queues generated by the electronic device are two, namely, the long frame queue (which can also be called the first image queue) and the short frame queue (which can also be called the second image queue). Among them, the long frame queue includes at least one first image (such as a long frame) collected by the electronic device under the first exposure duration, and the short frame queue includes at least one second image (such as a short frame) collected by the electronic device under the second exposure duration.

[0155] S507. The IFE module outputs a long frame based on the flag bit.

[0156] In the preview scenario, if there is banding, the AE module can output a flag bit to the IFE module. The IFE module does not fuse the long frame and the short frame according to this flag bit, and outputs the long frame to the camera service of the framework layer to display the long frame in the preview interface of the camera application. That is, the preview image displayed in the preview interface of the electronic device during the preview stage is a long frame.

[0157] S508. The perception module detects wonderful moments in real time, determines the timestamps corresponding to the wonderful moments, and sends the timestamps corresponding to the wonderful moments to the decision module.

[0158] After the automatic capture function or the manual capture function is turned on, the electronic device can identify the image frames collected by the image sensor in real time during the preview stage and determine the wonderful image frames. The moment corresponding to the wonderful image frame is the wonderful moment. Exemplarily, when the image sensor uses the dual-exposure mode to collect images, the perception module can determine the wonderful image frames according to the long frames collected by the image sensor.

[0159] In the automatic capture scenario or the manual capture scenario, the perception module of the electronic device can detect wonderful moments in real time during the preview stage. For example, the perception engine can detect parameters such as the state and / or actions of the shooting object, the exposure, and the clarity of each frame of the image in real time during the preview stage to determine whether the frame of the image is a wonderful image frame.

[0160] In some embodiments, algorithms can be preset in the perception module to identify wonderful moments. For example, the perception module can obtain the shooting object and the motion state of the object included in each frame of the image according to the preset image content extraction algorithm. When the shooting object is a person, the perception module can obtain the face and face attributes (such as whether the face is smiling, closing eyes, etc.) in the image through the above image content extraction algorithm. A clarity algorithm can also be preset in the perception module. According to this clarity algorithm, the perception module can obtain the clarity of each frame of the image. In some examples, when the image includes a person, the person is in the best completion state of a preset action, and the image is clear, the perception module can determine that the frame of the image is a wonderful image frame, and the moment corresponding to the wonderful image frame is the wonderful moment. In other examples, when the image includes a person, the person has open eyes and is smiling, and the image is clear, the perception module can determine that the frame of the image is a wonderful image frame.

[0161] In some examples, the perception module may obtain relevant data indicating the image content and image quality in the image frame, such as: the number, size, and position of the shooting objects, the completion degree of the actions, the number, size, and position of the faces, the smiling degree of the faces, the eye-opening degree of the faces, clarity, exposure, brightness, etc. Then, the excitement level (also referred to as the score) of the image frame is determined based on the above data. This excitement level is used to determine whether the image frame is exciting. The larger the value of the excitement level, the more exciting the image frame is. The acquisition moment corresponding to the exciting image is the exciting moment.

[0162] Exemplarily, the algorithms preset in the perception module may include a subject detection algorithm, an action evaluation algorithm, a face detection algorithm, a face attribute algorithm, and a clarity algorithm. Among them, the subject detection algorithm, the action evaluation algorithm, the face detection algorithm, and the face attribute algorithm can be referred to as image content extraction algorithms, which can be used to obtain image content. The clarity algorithm can be referred to as an image quality detection algorithm, which is used to obtain the image clarity.

[0163] The subject detection algorithm is used to detect the shooting objects in the image, such as people, cat faces, dogs, etc. The subject detection algorithm can also obtain the local features of the shooting objects, such as facial features. When the subject detection algorithm detects the shooting objects in the image, it can score the shooting objects according to their size and position in the image. The higher the score, the more centered the position of the shooting object.

[0164] The action evaluation algorithm is used to detect the actions of the shooting objects, such as jumping, looking back, throwing an object, playing table tennis, playing badminton, running, splashing water, playing baseball, playing football, flying a frisbee, etc., and determine the completion degree of the action in any frame of the image based on the best completion points of each action, and then obtain the action score of the image. The higher the action completion degree of the image, the higher the corresponding action score.

[0165] It can be understood that determining the best completion point of the action needs to be based on the running trajectory of the shooting object in multiple frames of images. Therefore, the action evaluation algorithm can, after receiving multiple frames of images, determine the best completion point of the shooting object in the multiple frames of images, and then evaluate the motion state of these multiple frames of images based on the best completion point.

[0166] For example, the perception module can analyze the motion changes of the shooting object in the preview image to identify whether the shooting object is in a motion state. When it detects that the shooting object is in a motion state, it can determine the moment corresponding to the image of the shooting moment according to the motion changes of the shooting object. For example, when the perception module identifies that the shooting object is jumping, it can determine the moment when the shooting object is in the air and the image is clear based on the posture of the shooting object after taking off as the exciting moment.

[0167] The face detection algorithm is used to detect faces in images. After a face is recognized, the face detection algorithm can evaluate the faces in the image according to the number, size, and position of the faces. For example, the larger the face and the more centered its position, the higher the score corresponding to that face.

[0168] The face attribute algorithm is used to detect face information in images, such as the curvature of the lips and the distance between the upper and lower eyelids. Based on the above-mentioned curvature of the lips and the distance between the upper and lower eyelids, it can be determined whether the face is open-eyed or smiling, etc. Further, the perception module can determine the degree of smiling of the face according to the curvature of the lips, and determine the degree of eye-opening of the face according to the distance between the upper and lower eyelids. Then, combined with the degree of smiling of the face and / or the degree of eye-opening of the face, the face performance is evaluated, and a face performance score is output. For example, the larger the curvature of the lips, the higher the degree of smiling of the corresponding face, and the higher the face performance score; the larger the distance between the upper and lower eyelids, the higher the degree of eye-opening of the face, and the higher the face performance score. When considering both the curvature of the lips and the distance between the upper and lower eyelids in a frame of image, the weights of the influence of eye-opening and smiling on the face performance score can be set, and the weights of the two can be the same or different.

[0169] In some examples, face detection can be performed after the subject detection recognizes a person, and the face attribute algorithm can be performed after the face detection algorithm recognizes a face. The face detection algorithm can obtain faces by using the image including people recognized by the subject detection. The face attribute algorithm can obtain the face information of each face by using the face image output by the face detection algorithm, thereby saving computational costs and improving algorithm efficiency.

[0170] The clarity algorithm is used to obtain the clarity of the image. The higher the clarity score of an image without jitter and noise.

[0171] In some examples, the perception module can determine an image quality score for each image frame by analyzing the attributes and features of the shooting scene and the image, such as color, brightness, contrast, clarity, exposure, etc. It can be understood that the perception module can also preset more detection algorithms to obtain more information indicating the image content and image quality.

[0172] In some embodiments, when determining the wonderful moment, the perception module can determine whether a frame of image is wonderful according to the data indicating the image content and image quality in a frame of image, or can obtain the data of one or more frames of images from a group of image frames to determine whether it is wonderful. For example, after the perception module determines a comprehensive score reflecting whether a frame of image is wonderful and the degree of wonderfulness for each image frame, it can determine the frame of image with the highest comprehensive score as the wonderful image frame, or directly determine whether the frame of image is a wonderful image frame according to the comprehensive score of each frame of image.

[0173] In some embodiments, to improve the accuracy of capturing and user satisfaction, the perception module can continuously optimize and improve its own algorithms and models through user feedback and learning mechanisms. By collecting user feedback on the capture results, the perception module can gradually learn the preferences and habits of users, so as to more accurately predict and capture wonderful moments that satisfy users. The embodiments of the present application do not limit the specific manner in which the perception module determines the wonderful moment, and the above embodiments are only illustrative descriptions.

[0174] When the perception module determines a wonderful image frame, it can determine the moment corresponding to the wonderful image frame as the wonderful moment, and output the timestamp corresponding to the wonderful moment to the decision module, so that the decision module can determine the sensor mode corresponding to the wonderful moment according to the timestamp corresponding to the wonderful moment.

[0175] S509. The decision module determines the sensor mode corresponding to the wonderful moment according to the timestamp corresponding to the wonderful moment, and sends the sensor mode corresponding to the wonderful moment to the frame selection module.

[0176] Exemplarily, when step S504 determines that there is no banding, the electronic device uses the single-exposure mode to collect images. At this time, the image frame queue only includes one image frame queue (such as the third image queue), and at least one frame of images collected with the exposure duration corresponding to the single-exposure mode (such as the first exposure duration) is included in the image frame queue. When step S504 determines that there is banding, the electronic device uses the dual-exposure mode to collect images. At this time, the image frame queue includes a long-frame queue and a short-frame queue. Therefore, after the perception module determines the timestamp corresponding to the wonderful moment, the decision module can determine whether the dual-exposure mode or the single-exposure mode is used at the wonderful moment according to the timestamp corresponding to the wonderful moment.

[0177] For example, when there is only one frame of image corresponding to the wonderful moment, it is determined that the sensor mode corresponding to the wonderful moment is the single-exposure mode; when there are two frames of images corresponding to the wonderful moment, it is determined that the sensor mode corresponding to the wonderful moment is the dual-exposure mode.

[0178] It should be noted that, in order to ensure that the sensor mode of the image sensor does not switch frequently, when detecting whether there is banding in step S504, the frequency threshold can be increased or the detection period can be set to ensure that the sensor mode used by the image sensor does not change within a short period of time. Therefore, after the decision module determines the sensor mode corresponding to the wonderful moment in step S509, when the frame selection module selects frames based on the wonderful moment, it can select the image frames collected at the wonderful moment and other moments near the wonderful moment, and the sensor mode corresponding to the other moment is the same as the sensor mode corresponding to the wonderful moment.

[0179] S510. The frame selection module selects frames in the image frame queue according to the sensor mode corresponding to the wonderful moment.

[0180] When the sensor mode is the dual-exposure mode, the image frame queue includes a long-frame queue and a short-frame queue. At this time, according to the time stamp corresponding to the highlight moment, the long frame and the short frame corresponding to the highlight moment can be selected from the long-frame queue and the short-frame queue, and at least one set of long and short frames corresponding to other moments near the highlight moment can be used as the target image frames. The target image frames are the image frames required to generate the captured image.

[0181] Exemplarily, the frame selection module can select one set of long and short frames corresponding to the highlight moment and one set of long and short frames corresponding to the next moment of the highlight moment, a total of two sets of long and short frames as the target image frames for generating the captured image.

[0182] For example, as Figure 11 shown, taking the highlight image frame as the 4th frame preview image (i.e., the preview image corresponding to the preview request 4) and two sets of image frames are required to generate the captured image as an example. Since the highlight image frame is the 4th frame preview image, the highlight moment is the acquisition moment of the 4th frame preview image. As Figure 11 shown in (a) of, when the highlight moment is the moment corresponding to the 4th frame preview image, the frame selection module can select the long frame 1 corresponding to the highlight moment and the long frame 2 corresponding to the next moment of the highlight moment (i.e., the moment corresponding to the 5th frame preview image) from the long-frame queue; then, according to the positions of the long frame 1 and the long frame 2 in the long-frame queue, the short frame 1 and the short frame 2 are respectively selected from the short-frame queue. Among them, the position of the short frame 1 in the short-frame queue is the same as the position of the long frame 1 in the long-frame queue, and the position of the short frame 2 in the short-frame queue is the same as the position of the long frame 2 in the long-frame queue. As Figure 11 shown in (a) of, the target image frames selected by the frame selection module include a set of long and short frames corresponding to the preview request 4 and a set of long and short frames corresponding to the preview request 5.

[0183] In some examples, in the case where the target image frames selected by the frame selection module include long frames and short frames, the multiple sets of long and short frames selected by the frame selection module can be stored in the sequence of long frame - short frame - long frame - short frame. Exemplarily, the first long frame and the first short frame can be the long frame and the short frame corresponding to the highlight moment.

[0184] For example, as Figure 11 shown in (a) of, the target image frames selected by the frame selection module include two sets of long and short frames, namely long frame 1, long frame 2, short frame 1 and short frame 2. Then, these two sets of long and short frames can be stored in the cache in the sequence of long frame 1 - short frame 1 - long frame 2 - short frame 2.

[0185] When the sensor mode is the single-exposure mode, the image frame queue only includes one queue. At this time, according to the time stamp corresponding to the highlight moment, the image frame corresponding to the highlight moment and the image frames corresponding to other moments near the highlight moment can be selected from the image frame queue as the target image frames.

[0186] Exemplarily, the frame selection module may select two image frames, i.e., the image frame corresponding to the wonderful moment and the image frame corresponding to the next moment of the wonderful moment, as the target image frames for generating the captured image.

[0187] For example, as shown in (b) of Figure 11 , when the image is acquired in the single-exposure mode, if the wonderful moment is the moment corresponding to the 4th frame preview image, the frame selection module may select the image frame 1 corresponding to the wonderful moment and the image frame 2 corresponding to the next moment of the wonderful moment (i.e., the moment corresponding to the 5th frame preview image) from the image frame queue. As shown in (b) of Figure 11 , the target image frames selected by the frame selection module include the image frame 1 corresponding to the preview request 4 and the image frame 2 corresponding to the preview request 5.

[0188] Exemplarily, the target image frames selected by the frame selection module may be stored in the buffer. After the capture instruction is triggered, the frame selection module may send the target image frames stored in the buffer to the capture algorithm module.

[0189] In some embodiments, in the case of banding, the image sensor may also acquire the image in the three-exposure mode. If the image sensor acquires the image in the three-exposure mode, there are three image frame queues, namely the long frame queue, the medium frame queue, and the short frame queue. At this time, the frame selection module may select a set of long and short frames corresponding to the wonderful moment and a set of long and short frames corresponding to the next moment of the wonderful moment, a total of two sets of long and short frames as the target image frames for generating the captured image; or may select a set of medium and short frames corresponding to the wonderful moment and a set of medium and short frames corresponding to the next moment of the wonderful moment, a total of two sets of frames as the target image frames for generating the captured image; or may select a set of long, medium, and short frames corresponding to the wonderful moment and a set of long, medium, and short frames corresponding to the next moment of the wonderful moment, a total of two sets of frames as the target image frames for generating the captured image. The frame selection method of the frame selection module in the embodiments of the present application is not limited, and only exemplary explanations are given here.

[0190] It should be noted that after the manual capture or automatic capture function is enabled, the perception module of the electronic device may detect the wonderful moment in real time during the preview phase. After the perception module determines the time stamp corresponding to the wonderful moment, the decision module will determine the sensor mode adopted by the electronic device at the wonderful moment according to the time stamp corresponding to the wonderful moment, and the frame selection module will perform frame selection according to the sensor mode and the time stamp corresponding to the wonderful moment.

[0191] In the manual capture scenario, through the preview function of the electronic device, the user can view whether the actions, expressions, angles, compositions, etc. of the captured object are optimal in the preview interface. When the user needs to capture a wonderful moment of the captured object, the user can click the photo control in the preview interface. Since the wonderful moment of the captured object usually occurs before the user clicks the photo control, in the capture scenario, the electronic device can perform frame selection during the preview stage, that is, the electronic device determines the wonderful moment during the preview stage and selects the target image frames required for synthesizing the captured image based on the wonderful moment. That is to say, the above steps S502 - S511 are all executed during the preview stage after the camera application is launched.

[0192] Exemplarily, the above decision module and frame selection module can be integrated into the photo taking module. In the automatic capture scenario, when the sensing module detects a wonderful moment, it will send a photo taking instruction to the photo taking module. In response to this photo taking instruction, the frame selection module in the photo taking module can send the selected target image frames to the photo taking algorithm module. It can be understood that, different from the manual photo taking scenario, in the automatic photo taking scenario, the photo taking instruction is triggered by the sensing module in the electronic device when it detects a wonderful moment.

[0193] S511. In response to the photo taking instruction, the frame selection module sends the selected frames to the photo taking algorithm module.

[0194] Exemplarily, the photo taking instruction can be triggered by the user or automatically triggered by the electronic device. In some examples, in the manual capture scenario, the user can trigger the photo taking instruction by inputting a photo taking operation in the preview interface, and the frame selection module sends the selected target image frames to the photo taking algorithm module based on the photo taking instruction triggered by the user. In other examples, in the automatic capture scenario, when the sensing module in the electronic device detects a wonderful moment, it can send a photo taking instruction to the photo taking module, and the frame selection module in the photo taking module sends the selected target image frames to the photo taking algorithm module based on the photo taking instruction.

[0195] For example, in the manual capture scenario, as Figure 12 shown, the user can see whether the jumping action of the captured object is in the best state in the preview interface 1201. When the user needs to capture the best jumping action of the captured object, the user can click the photo control 1202 in the preview interface 1201 (the user clicking the photo control 1202 in the preview interface is the user triggering the photo taking instruction). In response to the user clicking the photo control 1202, the frame selection module in the electronic device can send the target image frames it selects for generating the captured image to the photo taking algorithm module.

[0196] For another example, in the automatic capture scenario, when the sensing module determines a wonderful moment, it can send a photo taking instruction to the photo taking module, and the frame selection module in the photo taking module sends the selected target image frames to the photo taking algorithm module based on this photo taking instruction.

[0197] In some embodiments, in response to a photographing instruction, an electronic device may generate a photographing request and send the photographing request to a function module related to images in the electronic device. In response to the photographing request, the function module related to images in the electronic device may generate a captured image.

[0198] For example, in response to the photographing request, a frame selection module in the electronic device sends a target image frame to a photographing algorithm module, and the photographing algorithm module performs image processing based on the target image frame to generate a captured image.

[0199] S512. The photographing algorithm module performs fusion processing on the target image frame using a photographing algorithm to obtain a captured image.

[0200] Exemplarily, the photographing algorithm module may determine a corresponding photographing algorithm (such as the Motion Capture algorithm) according to the target image frame selected by the frame selection module, and perform fusion processing on the target image frame using the corresponding photographing algorithm to generate a captured image.

[0201] In some examples, when the target image frame includes a long frame and a short frame, the photographing algorithm module may use the short frame as a reference frame to fuse with the long frame to obtain a captured image. Exemplarily, when the target image frame includes multiple groups of long and short frames, the photographing algorithm module may use the first short frame (such as the third image) as a reference frame, and fuse other image frames (such as the fourth image) in the target image frame except the first short frame with the reference frame to generate a captured image.

[0202] For example, as shown in (a) of Figure 11 , the target image frames selected by the frame selection module are long frame 1, long frame 2, short frame 1, and short frame 2. Since these two groups of long and short frames are stored in the cache in the sequence of long frame 1 - short frame 1 - long frame 2 - short frame 2, after the photographing algorithm module obtains these 4 frames, it may use the first short frame 1 as a reference frame and then fuse it with long frame 1, long frame 2, and short frame 2.

[0203] It can be understood that since the exposure duration of the short frame is shorter and the exposure duration of the long frame is longer, the light and dark effect of the short frame is better and there is almost no banding phenomenon, and the image quality of the long frame is better. Therefore, when using the short frame as a reference frame to fuse with the long frame to generate a captured image, it can ensure that the generated captured image has no banding phenomenon and the quality of the generated captured image is better.

[0204] In some embodiments, in the dual-exposure mode, the target image frame includes multiple groups of long frames and short frames. Since the exposure durations of each group of long frames and short frames are different, the exposure amounts are also different. When fusing long frames and short frames with different exposure amounts, it will have a certain impact on the quality of the fused image. To further improve the quality of the captured image, when the image sensor captures images in the dual-exposure mode, the exposure amount of the short frames can be increased to ensure that the exposure amounts of the short frames and the long frames are the same.

[0205] Exemplarily, when the image sensor captures images in the dual-exposure mode, to ensure that the exposure amounts of the short frames and the long frames are the same, the image quality module PQ can send exposure configuration information (such as gain threshold) to the image sensor according to the configuration table, and the image sensor performs exposure according to this exposure configuration information to increase the gain value of the short frames. Thus, although the exposure durations of the long frames and short frames captured by the image sensor are different, the exposure amounts are the same.

[0206] In some embodiments, in the capture scenario, if the user enables the defocus function (such as background defocus), after step S512 above, steps S513 - S514 are further included.

[0207] S513. The photographing algorithm module sends the captured image to the defocus module.

[0208] S514. The defocus module generates a defocused captured image based on the captured image.

[0209] Exemplarily, when the defocus module generates a defocused captured image based on the captured image, the captured image can be used as the main path image, and then the auxiliary path image (such as the fifth image) is obtained from the auxiliary path queue (also referred to as the auxiliary path image queue). The main path image and the auxiliary path image can be images captured by different cameras in the electronic device. For example, the long frames and short frames of the captured image are image frames captured by one camera (such as the main camera) in the electronic device, while the images in the auxiliary path queue are image frames captured by another camera (such as the auxiliary camera) in the electronic device.

[0210] In some examples, the image corresponding to the exciting moment in the auxiliary path queue can be determined as the auxiliary path image. Since the main path image and the auxiliary path image are images captured by different cameras, there is a parallax between the main path image and the auxiliary path image (parallax refers to the position difference of the captured object in the two images when the two cameras photograph the same captured object). According to this parallax, the captured image can be defocused.

[0211] Exemplarily, generating a defocused captured image based on the main path image and the auxiliary path image includes: determining the parallax based on the main path image and the auxiliary path image, and defocusing the captured image based on this parallax to obtain a defocused captured image.

[0212] For example, as Figure 13As shown, the images in the long-frame queue and the short-frame queue can be captured by the main camera in the electronic device, and the images in the secondary queue can be captured by another camera (such as a secondary camera) in the electronic device. In step S512, the photographing algorithm module uses the first short frame 1 as a reference frame, and after fusing it with long frame 1, long frame 2, and short frame 2, a captured image can be obtained. The blurring module uses this captured image as the main road image, and uses the image frame 3 corresponding to the wonderful moment in the secondary queue as the secondary road image, calculates the parallax between the two frames of the main road image and the secondary road image, and then performs blurring processing on the captured image according to the parallax to obtain a blurred captured image.

[0213] The photographing method provided by the embodiment of the present application detects banding in the captured scene. In the case of the existence of banding, a dual-exposure mode is used to capture images. And in the preview stage, the long frame is sent for display while the short frame is not sent for display to ensure that the quality of the preview image seen by the user is better and the preview effect is improved. Moreover, since the screen refresh rate is relatively fast in the preview stage, the user will not see the banding phenomenon when the long frame is sent for display in the preview stage. In the photographing stage, by using the short frame as a reference frame to fuse with the long frame to generate a captured image, since the exposure duration of the short frame is short, the brightness and darkness effect of the short frame is good, and there is almost no banding phenomenon. Therefore, using the short frame as a reference frame to fuse with the long frame can obtain a captured image without banding, and the quality of the captured image is high.

[0214] Figure 14 It is a flowchart of another photographing method provided by the embodiment of the present application. As Figure 14 shown, the method includes the following steps S1401 - S1405.

[0215] S1401. Start the camera application in the electronic device.

[0216] Exemplarily, in response to a start instruction input by the user to the camera application of the electronic device, the camera application can be started. The embodiment of the present application will not elaborate on the start method of the camera application, and specific reference can be made to the foregoing step S501.

[0217] S1402. In response to a captured mode setting operation input by the user in the camera application, turn on the captured function and detect the banding phenomenon.

[0218] Exemplarily, in response to a capture mode setting operation input by a user in a camera application, the capture function is enabled, including: in response to a first operation input by the user on an auto-capture control in a preview interface, the auto-capture function is enabled; or, in response to a second operation input by the user on a manual-capture control in the preview interface, the manual-capture function is enabled; wherein, the capture mode setting operation includes the first operation and the second operation. It can be understood that the embodiments of the present application will not elaborate on the specific enabling manners of the auto-capture function and the manual-capture function, and for details, reference may be made to the foregoing step S502.

[0219] Exemplarily, detecting banding includes: obtaining the frequency of the light source where the electronic device is located; determining whether there is banding according to the frequency of the light source. When the frequency of the light source is greater than or equal to a frequency threshold, it is determined that there is banding; when the frequency of the light source is less than the frequency threshold, it is determined that there is no banding. It can be understood that the embodiments of the present application will not elaborate on the specific enabling manners of the auto-capture function and the manual-capture function, and for details, reference may be made to the foregoing steps S503-S504.

[0220] S1403. In the case of banding, collect an image based on a first sensor mode, and generate a first image queue and a second image queue.

[0221] At least one first image (such as a long frame) in the first image queue (such as a long-frame queue) is an image collected by the electronic device under a first exposure duration, and at least one second image (such as a short frame) in the second image queue (such as a short-frame queue) is an image collected by the electronic device under a second exposure duration, and the first exposure duration is greater than the second exposure duration.

[0222] S1404. Determine a highlight moment, and select a target image frame based on the highlight moment.

[0223] Exemplarily, selecting a target image frame based on the highlight moment includes: based on the highlight moment, determining the sensor mode corresponding to the highlight moment; the sensor mode includes a first sensor mode (such as a dual-exposure mode) and a second sensor mode (such as a single-exposure mode); when the sensor mode is the first sensor mode, select a target image frame from the first image queue and the second image queue.

[0224] In some embodiments, the dual-exposure mode may also be referred to as the dual-exposure high dynamic range (HDR) mode. HDR is a technology used to achieve a greater exposure dynamic range (i.e., a greater difference between light and dark) than ordinary digital image technologies. In the dual-exposure HDR mode, the image sensor can collect multiple frames of images based on different exposure durations to synthesize an image with a high dynamic range. Since the captured images in the embodiments of the present application are generated by fusing a long frame and a short frame with different exposure durations, the image quality of the captured images is relatively high.

[0225] It can be understood that the method for determining the wonderful moment and the method for selecting frames in the embodiments of the present application will not be described in detail, and reference may be specifically made to the foregoing steps S505-S510.

[0226] S1405. In response to the photographing instruction, use the third image in the target image frame as a reference frame and fuse it with the fourth image in the target image frame to generate a captured image.

[0227] Among them, the exposure duration of the third image is the second exposure duration, and the fourth image includes at least one image other than the third image in the target image frame. The fourth image may include a long frame and / or a short frame. That is, when generating the captured image, the short frame may be used as a reference frame and fused with the long frame to obtain the captured image.

[0228] In the case where the manual capture function is enabled, the photographing instruction is the photographing instruction input by the user on the preview interface; in the case where the automatic capture function is enabled, the photographing instruction is the instruction generated by the electronic device when detecting a wonderful moment.

[0229] It can be understood that the method for generating the captured image in the embodiments of the present application will not be described in detail, and reference may be specifically made to the foregoing steps S511-S512.

[0230] In some embodiments, when it is determined in the preview stage that there is banding, a preview image may be generated based on the first image and the preview image is displayed on the preview interface. That is, when it is determined in the preview stage that there is banding, the long frame is sent for display and the short frame is not sent for display to ensure better image quality in the preview stage and improve the preview effect.

[0231] Exemplarily, in the case where there is banding, the AE module outputs a flag bit to the IFE module to instruct the IFE module not to fuse the long frame and the short frame and directly output the long frame. For details, reference may be specifically made to the foregoing steps S505 and S507.

[0232] In some embodiments, the user can also enable the defocus function. The above method further includes: in the snapshot scenario, in response to the defocus enabling operation input by the user in the preview interface, enabling the defocus function; in the case where the defocus function is enabled, generating a defocused snapshot image based on the highlight moment and the snapshot image.

[0233] Exemplarily, generating a defocused snapshot image based on the highlight moment and the snapshot image includes: based on the highlight moment, determining a fifth image corresponding to the highlight moment in the auxiliary road image queue; wherein the images in the first image queue and the second image queue are images collected by the first camera of the electronic device, and the images in the auxiliary road image queue are images collected by the second camera of the electronic device, and the first camera is different from the second camera; determining the parallax based on the fifth image and the snapshot image; performing defocus processing on the snapshot image based on the parallax to obtain a defocused snapshot image.

[0234] It can be understood that the embodiments of the present application will not elaborate on the method of defocusing the snapshot image, and specific reference can be made to the foregoing steps S513 - S514.

[0235] In some embodiments, the above method further includes: in the case where there is no banding, collecting an image based on the second sensor mode and generating a third image queue; at least one frame of the images in the third image queue is an image collected by the electronic device under the first exposure duration.

[0236] Exemplarily, the second sensor mode can also be referred to as the single - exposure mode. The method of generating a preview image and a snapshot image in the single - exposure mode is the same as the prior art, and the embodiments of the present application do not limit this. It should be noted that since there is no banding when collecting images in the single - exposure mode, it is only necessary to continue to collect images in the single - exposure mode without switching the sensor mode. If banding is detected during the preview stage when collecting images in the single - exposure mode, the sensor mode is switched to the dual - exposure mode to ensure that the generated snapshot image has no banding.

[0237] It can be understood that in order to implement the above functions, the above - mentioned electronic device includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed in this article, the embodiments of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0238] Embodiments of the present application can divide the above-mentioned electronic device into functional modules according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present invention is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0239] Embodiments of the present application further provide a photographing device, which can be applied to an electronic device. The functions of the device can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as the photographing method provided in the foregoing embodiments.

[0240] Embodiments of the present application further provide an electronic device, which includes: a display screen, a memory, and one or more processors; the display screen and the memory are coupled to the processor; wherein, computer program code is stored in the memory, and the computer program code includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the photographing method provided in the foregoing embodiments. The specific structure of the electronic device can refer to Figure 4 the structure of the electronic device shown in

[0241] Embodiments of the present application further provide a computer-readable storage medium, which includes computer instructions. When the computer instructions run on an electronic device, the electronic device executes the photographing method provided in the foregoing embodiments.

[0242] Embodiments of the present application further provide a computer program product, which includes executable instructions. When the computer program product runs on an electronic device, the electronic device executes the photographing method provided in the foregoing embodiments.

[0243] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0244] In several embodiments provided in this application, it should be understood that the disclosed device / apparatus and method can be implemented in other ways. For example, the device / apparatus embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0245] The units described as separate components may or may not be physically separated. The components displayed as units can be one physical unit or multiple physical units, that is, they can be located in one place, or they can be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0246] In addition, in each embodiment of this application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0247] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical discs that can store program codes.

[0248] The above content is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A photographing method, characterized in that, Applied to an electronic device, the method includes: Launch the camera application in the electronic device; In response to a capture mode setting operation input by the user in the camera application, enable the capture function and detect the banding phenomenon; In the presence of banding, acquire images based on a first sensor mode and generate a first image queue and a second image queue; at least one first image in the first image queue is an image acquired by the electronic device at a first exposure duration, and at least one second image in the second image queue is an image acquired by the electronic device at a second exposure duration, where the first exposure duration is greater than the second exposure duration; Determine a wonderful moment and select a target image frame based on the wonderful moment; In response to a photographing instruction, use a third image in the target image frame as a reference frame and fuse it with a fourth image in the target image frame to generate a captured image; where the exposure duration of the third image is the second exposure duration, and the fourth image includes images in the target image frame other than the third image.

2. The method according to claim 1, wherein The detecting the banding phenomenon includes: Obtain the frequency of the light source where the electronic device is located; Determine whether there is banding according to the frequency of the light source.

3. The method according to claim 2, wherein The determining whether there is banding according to the frequency of the light source includes: Determine that there is banding when the frequency of the light source is greater than or equal to a frequency threshold; Determine that there is no banding when the frequency of the light source is less than the frequency threshold.

4. The method according to claim 1, wherein The method further includes: In the presence of banding, generate a preview image based on the first image; Display the preview image on a preview interface.

5. The method according to any one of claims 1-4, characterized in that, The selecting the target image frame based on the wonderful moment includes: Based on the wonderful moment, determine the sensor mode corresponding to the wonderful moment; the sensor mode includes the first sensor mode and the second sensor mode; When the sensor mode is the first sensor mode, select the target image frame from the first image queue and the second image queue.

6. The method according to any one of claims 1-4, characterized in that, The enabling the capture function in response to a capture mode setting operation input by the user in the camera application includes: In response to a first operation input by the user to an automatic capture control on the preview interface, enable the automatic capture function; or, In response to a second operation input by the user to a manual capture control on the preview interface, enable the manual capture function; Wherein, the capture mode setting operation includes the first operation and the second operation.

7. The method according to claim 6, wherein When the manual capture function is enabled, the photographing instruction is an instruction input by the user on the preview interface; when the automatic capture function is enabled, the photographing instruction is an instruction generated by the electronic device when the wonderful moment is detected.

8. The method according to any one of claims 1-4, characterized in that, The method further includes: In response to a defocusing enabling operation input by the user on the preview interface, enable the defocusing function; When the defocusing function is enabled, generate a defocused captured image based on the wonderful moment and the captured image.

9. The method according to claim 8, wherein Generating a blurred captured image based on the wonderful moment and the captured image includes: Based on the wonderful moment, determining a fifth image corresponding to the wonderful moment in the auxiliary road image queue; wherein, the images in the first image queue and the second image queue are images captured by a first camera of the electronic device, the images in the auxiliary road image queue are images captured by a second camera of the electronic device, and the first camera is different from the second camera; Determining a parallax based on the fifth image and the captured image; Performing a blurring process on the captured image based on the parallax to obtain the blurred captured image.

10. The method according to any one of claims 1-4, characterized in that The method further includes: In the absence of banding, collecting images based on a second sensor mode and generating a third image queue; at least one frame of the images in the third image queue is an image captured by the electronic device under the first exposure duration.

11. An electronic device, characterized in that, Including: A display screen, a memory, and one or more processors; the display screen and the memory are coupled to the processor; wherein, computer program code is stored in the memory, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device is caused to execute the photographing method according to any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, Including computer instructions, and when the computer instructions are run on an electronic device, the electronic device is caused to execute the photographing method according to any one of claims 1-10.

Images