Frame preview method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a frame preview method and device, electronic equipment and a storage medium, a camera sensor is set by using sensor exposure parameters in response to adjustment of a picture output mode of the camera sensor from a first mode to a second mode, and the sensor exposure parameters comprise a normal exposure parameter and a variable exposure parameter; obtaining a normal exposure frame collected by the camera sensor under the normal exposure parameter and a variable exposure frame collected under the variable exposure parameter; sending the variable exposure frame into a photographing path; and previewing and displaying the normal exposure frame. Therefore, the image frame can be previewed under the condition of variable exposure photographing, and the preview experience of the user is improved due to the fact that the previewed frame is the normal exposure frame.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a frame preview method, device, electronic device and storage medium. Background Art

[0002] Backlight and high-contrast environments are high-dynamic scenes. In high-dynamic scenes, variable exposure photography is often used to ensure the dynamic range and effect of the photos. Variable exposure photography is to adjust the exposure time of the camera sensor through the automatic exposure module (automatic exposure, AE) to generate image frames with different exposures. However, when previewing image frames in variable exposure photography, due to the different exposures of the image frames, the brightness changes of individual image frames will be particularly obvious, and users will see the phenomenon of brightness jumps in the image frames. In order to ensure a good preview experience for users, these variable exposure frames will be discarded and not previewed. In this way, after the user clicks the shutter control, there will be a short preview freeze in the preview area on the display.

[0003] As for LivePhoto, it consists of a short preview clip and a photo. In order to ensure the smoothness of the preview clip, the variable exposure strategy is not allowed when taking Live Photos. If the variable exposure strategy is used, the corresponding variable exposure frame will be discarded when sending the preview, thus affecting the smoothness of the preview clip. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a frame preview method, device, electronic device and storage medium to solve the problem of poor preview effect when taking photos with variable exposure. The specific technical solution is as follows:

[0005] In a first aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application provides a frame preview method, the method comprising:

[0006] In response to the image output mode of the camera sensor being adjusted from the first mode to the second mode, setting the camera sensor using a sensor exposure parameter, wherein the sensor exposure parameter includes a normal exposure parameter and a variable exposure parameter;

[0007] Acquire a normal exposure frame acquired by the camera sensor under the normal exposure parameter, and a variable exposure frame acquired by the camera sensor under the variable exposure parameter;

[0008] Sending the variable exposure frame into a photographing path;

[0009] The preview displays the normal exposure frame.

[0010] In the embodiment of the present application, in response to the camera sensor's image output mode being adjusted from the first mode to the second mode, the camera sensor is set using sensor exposure parameters, wherein the sensor exposure parameters include normal exposure parameters and variable exposure parameters; the normal exposure frame collected by the camera sensor under the normal exposure parameters and the variable exposure frame collected under the variable exposure parameters are obtained; the variable exposure frame is sent to the photo-taking path; and the normal exposure frame is previewed and displayed. This achieves that the image frame can be previewed even in the case of variable exposure photo-taking, and since the previewed frame is a normal exposure frame, the user's preview experience is also improved.

[0011] In a possible implementation, the method further includes:

[0012] When the first mode is the merge mode, a first exposure parameter of the camera sensor in the merge mode is acquired to obtain a normal exposure parameter.

[0013] In an embodiment of the present application, for the case where the first mode is a merge mode, the first exposure parameter of the camera sensor in the merge mode is obtained. In this way, when the camera sensor mode is switched to the second mode, the exposure parameter of the camera sensor can be set with the first exposure parameter and the variable exposure parameter.

[0014] In a possible implementation, the method further includes:

[0015] When the first mode is a dual-gain conversion mode, a short exposure frame acquired by the camera sensor in a low-gain conversion mode and a long exposure frame acquired in a high-gain conversion mode are acquired; wherein the dual-gain conversion mode includes a low-gain conversion mode and a high-gain conversion mode;

[0016] Merging the short exposure frame and the long exposure frame;

[0017] A second exposure parameter of the fused exposure frame is determined to obtain a normal exposure parameter.

[0018] In an embodiment of the present application, for the case where the first mode is a dual gain conversion mode, a short exposure frame collected by the camera sensor in a low gain conversion mode and a long exposure frame collected in a high gain conversion mode are obtained, the short exposure frame and the long exposure frame are fused, and the exposure parameters of the fused exposure frame are determined, that is, the second exposure parameters of the camera sensor in the dual gain conversion mode, to obtain normal exposure parameters. In this way, when the camera sensor mode is switched to the second mode, the exposure parameters of the camera sensor can be set with the second exposure parameters and the variable exposure parameters.

[0019] In a possible implementation, the method further includes:

[0020] When the first mode is the dual gain conversion mode, a third exposure parameter of the camera sensor in the low gain conversion mode and a fourth exposure parameter in the high gain conversion mode are obtained; wherein the dual gain conversion mode includes a low gain conversion mode and a high gain conversion mode, and the normal exposure parameters include a third exposure parameter and a fourth exposure parameter.

[0021] In an embodiment of the present application, when the first mode is the dual gain conversion mode, the image output mode of the camera sensor is to output two frames at the same time, namely, a short exposure frame corresponding to the low gain conversion mode and a long exposure frame corresponding to the high gain conversion mode. At this time, it is necessary to obtain the third exposure parameter of the camera sensor in the low gain conversion mode and the fourth exposure parameter in the high gain conversion mode. In this way, when the camera sensor mode is switched to the second mode, the exposure parameter of the camera sensor can be set with the third exposure parameter, the fourth exposure parameter and the variable exposure parameter.

[0022] In a possible implementation manner, obtaining the normal exposure frame collected by the camera sensor under the normal exposure parameters includes:

[0023] Acquire a short exposure frame acquired by the camera sensor under a third exposure parameter and a long exposure frame acquired under a fourth exposure parameter;

[0024] The preview displays the normal exposure frame, including:

[0025] Merging the short exposure frame and the long exposure frame;

[0026] The preview shows the blended frame.

[0027] In the embodiment of the present application, for the case where the first mode is a dual gain conversion mode, short exposure frames are captured using the third exposure parameters in the low gain conversion mode, and long exposure frames are captured using the fourth exposure parameters in the high gain conversion mode. The consistency of the preview effect can be ensured by the short exposure frames and the long exposure frames.

[0028] In a possible implementation, the method further includes:

[0029] After a preset number of variable exposure frames are acquired, the image output mode of the camera sensor is triggered to be adjusted from the second mode to the first mode;

[0030] Restoring the exposure parameters of the camera sensor to the corresponding exposure parameters in the first mode;

[0031] The camera sensor is driven to capture image frames with corresponding exposure parameters in the first mode.

[0032] In the embodiment of the present application, after the variable exposure photography is completed, the image output mode of the camera sensor is adjusted from the second mode to the first mode, so that the camera sensor still captures image frames with the corresponding exposure parameters in the first mode.

[0033] In a possible implementation, the second mode is an interlaced high dynamic range imaging mode.

[0034] In a possible implementation, the method further includes:

[0035] After obtaining a shooting request for variable exposure photography, the image output mode of the camera sensor is triggered to be adjusted from the first mode to the second mode.

[0036] In a second aspect, an embodiment of the present application provides a frame preview device, the device comprising:

[0037] an exposure parameter setting module, configured to, in response to the image output mode of the camera sensor being adjusted from the first mode to the second mode, set the camera sensor using a sensor exposure parameter, wherein the sensor exposure parameter includes a normal exposure parameter and a variable exposure parameter;

[0038] An acquisition module, used to acquire the normal exposure frame collected by the camera sensor under the normal exposure parameter, and the variable exposure frame collected by the camera sensor under the variable exposure parameter;

[0039] A photographing module, used for sending the variable exposure frame into a photographing path;

[0040] The preview module is used to preview and display the normal exposure frame.

[0041] In a possible implementation, the device further includes:

[0042] The first exposure parameter acquisition module is used to acquire the first exposure parameter of the camera sensor in the merge mode to obtain a normal exposure parameter when the first mode is the merge mode.

[0043] In a possible implementation, the device further includes:

[0044] A second exposure parameter acquisition module, used for acquiring short exposure frames acquired by the camera sensor in a low gain conversion mode and long exposure frames acquired in a high gain conversion mode when the first mode is a dual gain conversion mode; wherein the dual gain conversion mode includes a low gain conversion mode and a high gain conversion mode;

[0045] Merging the short exposure frame and the long exposure frame;

[0046] A second exposure parameter of the fused exposure frame is determined to obtain a normal exposure parameter.

[0047] In a possible implementation, the device further includes:

[0048] The third and fourth exposure parameter acquisition modules are used to obtain the third exposure parameter of the camera sensor in the low gain conversion mode and the fourth exposure parameter in the high gain conversion mode when the first mode is the dual gain conversion mode; wherein the dual gain conversion mode includes the low gain conversion mode and the high gain conversion mode, and the normal exposure parameters include the third exposure parameter and the fourth exposure parameter.

[0049] In a possible implementation manner, the acquisition module is specifically used to:

[0050] Acquire a short exposure frame acquired by the camera sensor under a third exposure parameter and a long exposure frame acquired under a fourth exposure parameter;

[0051] The preview module is specifically used to fuse the short exposure frame and the long exposure frame; and preview and display the fused frame.

[0052] In a possible implementation, the device further includes:

[0053] An image output mode recovery module, used to trigger the image output mode of the camera sensor to be adjusted from the second mode to the first mode after a preset number of variable exposure frames are acquired;

[0054] An exposure parameter recovery module, used to recover the exposure parameters of the camera sensor to the corresponding exposure parameters in the first mode;

[0055] The acquisition module is used to drive the camera sensor to acquire image frames with corresponding exposure parameters in the first mode.

[0056] In a possible implementation, the second mode is an interlaced high dynamic range imaging mode.

[0057] In a possible implementation, the device further includes:

[0058] The image output mode adjustment module is used to trigger the image output mode of the camera sensor to be adjusted from the first mode to the second mode after obtaining a shooting request for variable exposure photography.

[0059] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0060] one or more processors and memory;

[0061] The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute any frame preview method described in the first aspect.

[0062] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, comprising a computer program, which, when executed on an electronic device, enables the electronic device to execute any of the frame preview methods described in the first aspect.

[0063] The frame preview method provided in the embodiment of the present application, in response to the camera sensor's image output mode being adjusted from the first mode to the second mode, uses the sensor exposure parameters to set the camera sensor, wherein the sensor exposure parameters include normal exposure parameters and variable exposure parameters; obtains the normal exposure frame collected by the camera sensor under the normal exposure parameters, and the variable exposure frame collected under the variable exposure parameters; sends the variable exposure frame to the photo-taking path; and previews and displays the normal exposure frame. This achieves that the image frame can be previewed even in the case of variable exposure photo-taking, and because the previewed frame is a normal exposure frame, the user's preview experience is also improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0065] Figure 1 A hardware structure diagram of an electronic device provided in an embodiment of the present application;

[0066] Figure 2 A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0067] Figure 3 A schematic diagram of a user interface provided in an embodiment of the present application;

[0068] Figure 4 A schematic diagram of a shooting interface provided in an embodiment of the present application;

[0069] Figure 5a A schematic diagram of a set of exposure frames with different exposure parameters provided in an embodiment of the present application;

[0070] Figure 5b A set of preview images displayed after being processed by the processing flow of the prior art;

[0071] Figure 5c A set of preview screens displayed after being processed using the processing flow of this application;

[0072] Figure 6 A flowchart of a frame preview method provided in an embodiment of the present application;

[0073] Figure 7 A frame preview method in a merge mode provided in an embodiment of the present application;

[0074] Figure 8 A frame preview method in a dual gain conversion mode provided by an embodiment of the present application;

[0075] Fig. 9 A frame preview method provided in an embodiment of the present application;

[0076] Fig.10 A schematic diagram of the structure of a frame preview device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0078] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first instruction and the second instruction are to distinguish different user instructions, and their order is not limited. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit them to be different.

[0079] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0080] The following describes some concepts involved in the frame preview method provided in the embodiment of the present application.

[0081] Variable exposure frame: that is, abnormal exposure frame, including long exposure frame and short exposure frame;

[0082] Long exposure frame: exposure frame with EV greater than EV0;

[0083] Short exposure frame: exposure frame with EV less than EV0;

[0084] EV: A quantity that reflects the amount of exposure, where EV0 refers to the exposure obtained when ISO (International Organization for Standardization) is 100, the aperture coefficient is F1, and the exposure time is 1S.

[0085] Variable exposure shooting mode: refers to a shooting mode with HDR (High Dynamic Range) turned on. It is a shooting mode that collects image frames with multiple exposure parameters such as long exposure frames, short exposure frames and normal exposure frames, and fuses the image frames with multiple exposure parameters to obtain a high dynamic range image.

[0086] Backlight and high-contrast environments are high-dynamic scenes. In high-dynamic scenes, variable exposure photography is often used to ensure the dynamic range and effect of the photos. Variable exposure photography is to adjust the exposure time of the camera sensor through the automatic exposure module (automatic exposure, AE) to generate image frames with different exposures. However, when previewing image frames in variable exposure photography, due to the different exposures of the image frames, the brightness changes of individual image frames will be particularly obvious, and users will see the phenomenon of brightness jumps in the image frames. In order to ensure a good preview experience for users, these variable exposure frames will be discarded and not previewed. In this way, after the user clicks the camera's shooting control, there will be a short preview freeze in the preview area on the display.

[0087] As for live photos, they consist of a short preview clip and a photo. The generation of the photo is consistent with the normal photo process. In order to ensure the smoothness of the preview clip, the variable exposure strategy is not allowed when taking live photos. If the variable exposure strategy is used, the corresponding variable exposure frame will be discarded when sending the preview, thus affecting the smoothness of the preview clip. In this way, when taking live photos, only non-variable exposure photography can be used, but non-variable exposure photography cannot achieve good photography effects in high-dynamic scenes. The photos taken by live photos in high-dynamic scenes are not the best.

[0088] In order to solve the problem of poor preview effect when taking photos with variable exposure, the frame preview method provided in the embodiment of the present application is introduced below.

[0089] The embodiment of the present application provides a frame preview method, which can be applied to electronic devices. The electronic device is an electronic device with a camera and provides photo taking and display services, such as a smart phone, a tablet computer, a smart watch, etc. When variable exposure photo taking, the image output mode of the camera sensor is adjusted from the first mode to the second mode, and in response to the image output mode of the camera sensor being adjusted from the first mode to the second mode, the exposure parameters of the camera sensor are set using normal exposure parameters and variable exposure parameters, and the normal exposure frame collected by the camera sensor under the normal exposure parameters and the variable exposure frame collected under the variable exposure parameters are obtained; then the variable exposure frame is sent to the photo taking path to generate a photo, and the normal exposure frame is sent to the preview path to preview and display the normal exposure frame. When the image output mode of the camera sensor is switched to the second mode, the camera sensor will simultaneously output the normal exposure frame and the variable exposure frame, and send the normal exposure frame to the preview display. Since the exposure of each image frame is consistent when the user previews the image frame, it can be seen that the method of the embodiment of the present application solves the problem of poor preview effect when variable exposure photo taking.

[0090] The structure of the electronic device to which the above frame preview method is applied is introduced below.

[0091] For example, Figure 1 The structure diagram of the electronic device 100 is shown. The electronic device 100 may include a processor 110, a display screen 120, a camera 130, an internal memory 140, a SIM (Subscriber Identification Module) card interface 150, a USB (Universal Serial Bus) interface 160, a charging management module 170, a battery management module 171, a battery 172 with a cell and a battery protection device, a sensor module 180, a mobile communication module 190, a wireless communication module 200, an antenna 1 and an antenna 2, etc. The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, an ambient light sensor 180D, etc.

[0092] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0093] The processor 110 may include one or more processing units, for example: the processor 110 may include a CPU (Central Processing Unit), an AP (Application Processor), a modem processor, a GPU (graphics processing unit), an ISP (Image Signal Processor), a controller, a video codec, a DSP (Digital Signal Processor), a baseband processor, and / or an NPU (Neural-network Processing Unit), etc. Among them, different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110. Among them, the controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. In some other embodiments, a memory may also be provided in the processor 110 for storing instructions and data. Exemplarily, the memory in the processor 110 may be a high-speed cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the electronic device 100 in processing data or executing instructions.

[0094] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an I2C (Inter-Integrated Circuit) interface, an I2S (Inter-Integrated Circuit Sound) interface, a PCM (Pulse Code Modulation) interface, a UART (Universal Asynchronous Receiver / Transmitter) interface, a MIPI interface (Mobile Industry Processor Interface), a GPIO (General-Purpose Input / Output) interface, a SIM card interface and / or a USB interface, etc. Among them, the USB interface 160 is an interface that complies with the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 160 may be used to connect a charger to charge the electronic device 100, and may also be used to transmit data between the electronic device 100 and a peripheral device. The USB interface 160 may also be used to connect headphones to play audio through the headphones.

[0095] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is for illustrative purposes only and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0096] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 190, the wireless communication module 200, the modem processor, the baseband processor, and the like.

[0097] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0098] The electronic device 100 implements the display function through a GPU, a display screen 120, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 120 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0099] The display screen 120 is used to display images, videos, etc. The display screen 120 includes a display panel. The display panel can be LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), active matrix organic light emitting diode or AMOLED (Active-Matrix Organic Light-Emitting Diode), FLED (Flex Light-Emitting Diode), MiniLed, MicroLed, Micro-oLed, QLED (Quantum Dot Light Emitting Diodes), etc. In some embodiments, the electronic device 100 may include one or more display screens 120.

[0100] In some embodiments of the present application, when the display panel is made of materials such as OLED, AMOLED, FLED, etc., the above Figure 1 The display screen 120 in the embodiment can be bent. Here, the display screen 120 can be bent means that the display screen can be bent to any angle at any position and can be kept at the angle. For example, the display screen 120 can be folded in half from the middle to the left or right. It can also be folded in half from the middle to the top or bottom.

[0101] The display screen 120 of the electronic device 100 can be a flexible screen. At present, flexible screens have attracted much attention for their unique characteristics and huge potential. Compared with traditional screens, flexible screens have the characteristics of strong flexibility and bendability, which can provide users with a new interaction method based on bendable characteristics, and can meet users' more needs for electronic devices. For electronic devices equipped with a foldable display, the foldable display on the electronic device can be switched between a small screen in a folded form and a large screen in an unfolded form at any time. Therefore, users use the split-screen function on electronic devices equipped with a foldable display more and more frequently.

[0102] The electronic device 100 can realize the shooting function through the ISP, the camera 130, the video codec, the GPU, the display screen 120 and the application processor, etc., wherein the camera 130 includes a front camera and a rear camera.

[0103] The ISP is used to process the data fed back by the camera 130. For example, when shooting, the shutter is opened, and light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can perform algorithm optimization on the noise, brightness and color of the image. The ISP can also optimize the exposure and color temperature of the shooting scene and other parameters. In some embodiments, the ISP can be set in the camera 130.

[0104] The camera 130 is used to take photos or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a CCD (Charge Coupled Cevice) or a CMOS (Complementary Metal-Oxide-Semiconductor) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB (Red Green Blue), YUV (a color encoding method) or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 130, where N is a positive integer greater than 1.

[0105] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0106] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as MPEG (Moving Picture Experts Group) 1, MPEG2, MPEG3, and MPEG4.

[0107] NPU is a neural network computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0108] The internal memory 140 can be used to store one or more computer programs, which include instructions. The processor 110 can enable the electronic device 100 to perform the frame preview method provided in some embodiments of the present application, as well as various applications and data processing, etc. by running the above instructions stored in the internal memory 140. The internal memory 140 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 100. In addition, the internal memory 140 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, flash memory components, general flash memory, etc. In some embodiments, the processor 110 can enable the electronic device 100 to perform the frame preview method provided in the embodiment of the present application, as well as other applications and data processing by running instructions stored in the internal memory 140 and / or instructions stored in a memory provided in the processor 110.

[0109] The internal memory 140 can be used to store the relevant programs of the frame preview method provided in the embodiment of the present application, and the processor 110 can be used to call the relevant programs of the frame preview method stored in the internal memory 140 when displaying information to execute the frame preview method of the embodiment of the present application.

[0110] The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, an ambient light sensor 180D, and the like.

[0111] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 120. There are many types of pressure sensors 180A, for example, they can be resistive pressure sensors, inductive pressure sensors or capacitive pressure sensors. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When the force acts on the pressure sensor 180A, the capacitance between the electrodes changes, and the electronic device 100 determines the intensity of the pressure according to the change in capacitance. When the touch operation acts on the display screen 120, the electronic device 100 detects the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on a short message application icon, an instruction to view the short message is executed; when a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on a short message application icon, an instruction to create a new short message is executed.

[0112] The fingerprint sensor 180B is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0113] The touch sensor 180C is also called a touch control device. The touch sensor 180C can be arranged on the display screen 120, and the touch sensor 180C and the display screen 120 form a touch screen, which is also called a touch control screen. The touch sensor 180C is used to detect a touch operation acting on or near it. The touch sensor 180C can pass the detected touch operation to the application processor to determine the type of touch event. A visual output related to the touch operation can be provided through the display screen 120. In other embodiments, the touch sensor 180C can also be arranged on the surface of the electronic device 100, and is arranged at a different position from the display screen 120.

[0114] The ambient light sensor 180D is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 120 according to the perceived brightness of the ambient light. The ambient light sensor 180D can also be used to automatically adjust the white balance when shooting. The ambient light sensor 180D can also transmit the environmental information of the device to the GPU.

[0115] The ambient light sensor 180D is also used to obtain the brightness, light ratio, color temperature, etc. of the environment in which the camera 130 captures images.

[0116] Figure 2A software structure block diagram of an electronic device to which the present application is applicable. The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The layered architecture divides the software system of the electronic device into several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces. In some embodiments, the software system can be divided into five layers, namely, an application layer, an application framework layer, a system operation layer, a HAL (Hardware Abstract Layer) and a kernel layer.

[0117] The application layer may include a camera, a gallery, and third-party applications with camera functions, etc. Exemplarily, the application layer may also include applications such as calls, calendars, maps, navigation, music, videos, and short messages.

[0118] The application framework layer provides application programming interface and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0119] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, a camera service, and the like.

[0120] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0121] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0122] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0123] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0124] The notification manager allows applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, notification information is used to inform the completion of downloads, message reminders, etc. Notification information can also appear in the system top status bar in the form of icons or scroll bar text, such as notifications of applications running in the background, or in the form of dialog windows on the screen. Notification information can be, for example, a text message in the status bar, a reminder sound, vibration of electronic equipment, flashing indicator lights, etc.

[0125] The camera service is used to call the camera (including the front camera and / or the rear camera) in response to the application's request. Specifically in the embodiment of the present application, the camera service can call the camera to collect images in response to the photo command to obtain the corresponding image frame. For example, it is detected that the user clicks the camera's shooting control, determines that the photo command is received, and calls the camera to collect images.

[0126] The system operation layer may include multiple functional modules, such as a surface manager, a media library, a 3D (Three Dimensional) graphics processing library (such as OpenGL ES), a 2D (Two Dimensional) graphics engine, and the like.

[0127] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0128] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264 (a new generation of digital video compression format), MP3 (Moving Picture Experts Group Audio Layer III, dynamic image expert compression standard audio layer 3), AAC (Advanced Audio Coding, advanced audio coding), AMR (Adaptive Multi-Rate, adaptive multi-rate), JPG (Joint Photographic Experts Group, an image file format), PNG (Portable Network Graphics, portable network graphics), etc.

[0129] The 3D graphics processing library is used to implement three-dimensional graphics drawing, image rendering, compositing, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0130] HAL is an interface layer between the operating system kernel and the hardware circuit. HAL includes but is not limited to: an image processing module. Among them, the image processing module is used to process the image stream. In the embodiment of the present application, after the camera is started, the image processing module can receive the image stream from the camera and can process the image stream, such as noise reduction processing, brightness processing, etc.

[0131] Exemplarily, when the camera application is running, the camera service can send the user-triggered photo command to the camera service. On the one hand, the camera service can send the received photo command to the image processing module, so that the image processing module can call the camera driver in the kernel layer according to the received photo command, and the camera driver drives the camera and other hardware devices to respond to the photo command to collect image frames. For example, the camera can pass each frame of the collected image to the image processing module at a certain frame rate, and the image processing module will process it and send it to the display screen for display. On the other hand, after receiving the above-mentioned photo command, the camera service can determine the shooting strategy at this time according to the received photo command, and the shooting strategy sets the specific image processing tasks that need to be performed on the collected image data. For example, in the preview mode, the photo command can set the image processing task 1 in the shooting strategy to realize the face detection function. For another example, if the user turns on the beauty function in the preview mode, the camera service can also set the image processing task 2 in the shooting strategy to realize the beauty function. Then, the camera service can send the determined shooting strategy to the image processing module. When the image processing module receives each frame of image data collected by the camera, it can perform corresponding image processing tasks on the above image data according to the shooting strategy issued by the camera service to obtain each frame of the shooting picture after image processing.

[0132] Subsequently, the image processing module may report each frame of the captured image after image processing to the camera application through the camera service, and the camera application may display each frame of the captured image in the preview area of ​​the display screen, or the camera application may save each frame of the captured image in the electronic device in the form of a photo or video.

[0133] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, image sensor driver, audio driver, etc. The hardware includes at least sensors, display screens, cameras, ISP, etc.

[0134] Understandably, Figure 2 The layers in the software structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0135] It is understandable that in order to implement the frame preview method provided in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0136] An application scenario of the frame preview method provided in the embodiment of the present application is described below. In this scenario, an electronic device is taken as an example of a mobile phone for explanation.

[0137] Figure 3 The user interface 300 for displaying applications on a mobile phone is shown as an example. The user interface 300 displays a page with application icons, which may include multiple application icons (for example, weather application icon, calendar application icon, mail application icon, settings application icon, application store application icon, notes application icon, album application icon, etc.). A page indicator may also be displayed below the above multiple application icons to indicate the positional relationship between the currently displayed page and other pages. There are multiple tray icons (for example, camera application icon 310, browser application icon, phone application icon, information application icon) below the page indicator. The tray icon remains displayed when the page is switched. The embodiment of the present application does not limit the content displayed on the user interface 300.

[0138] The mobile phone can detect a user operation (such as a touch / click operation) on the camera application icon 310, and in response to the operation, the mobile phone can display Figure 4 The shooting interface 400 shown. The shooting interface 400 can be the user interface of the default shooting mode of the camera application, and the user can complete the photo taking on this interface. The camera application is an image shooting application on electronic devices such as smart phones and tablets. This application does not limit the name of the application. In other words, the user can open the shooting interface 400 of the camera application by clicking the camera application icon 310. It can be understood that the default camera in the default shooting mode is not limited to the rear camera, and the mobile phone can also set the front camera as the default camera. In other words, after turning on the camera application, the mobile phone can display the image captured by the front camera in the preview area 420, which can be used for the user to take pictures through the default front camera.

[0139] Figure 4 The photographing interface 400 of the camera application in the mobile phone is exemplarily shown. Figure 4As shown, the shooting interface 400 may include a parameter adjustment area 410, a preview area 420, a camera mode option area 430, a gallery shortcut control 441, a shutter control 442, and a camera flip control 443. Among them, each control in the parameter adjustment area 410 is used to adjust the corresponding shooting parameters, including but not limited to: a flash setting control, an HDR switch setting control 411, and a more detailed camera setting control. The HDR switch setting control 411 can be used to trigger the electronic device to use the HDR algorithm to fuse multiple images with different exposure parameters when it is turned on.

[0140] The preview area 420 can be used to display a preview image, which is an image collected by the mobile phone through the camera in real time. The mobile phone can refresh the display content in the preview area 420 in real time to facilitate the user to preview the image currently collected by the camera.

[0141] One or more shooting mode options may be displayed in the camera mode option area 430. The one or more shooting mode options may include: night scene mode option 431, portrait mode option 432, photo mode option 433, video mode option 434, and more options 435. It can be understood that the one or more shooting mode options may be displayed as text information on the interface, such as "night scene", "portrait", "photo", "video", "more", or as icons or other forms of interactive elements (IE), which are not limited in the present application.

[0142] When a user operation on a shooting mode option is detected, the mobile phone can start the shooting mode selected by the user. In particular, when a user operation on more options 435 is detected, the mobile phone can further display more other shooting mode options, such as time-lapse photography mode options, continuous shooting mode options, etc., to show the user more abundant camera functions. It is understandable that the camera mode option area 430 can also include more or fewer shooting mode options. Figure 4 The camera mode options shown are only one implementation of the present application and should not be considered as a limitation of the present application.

[0143] The gallery shortcut control 441 can be used to open the gallery application. In response to a user operation, such as a touch operation, acting on the gallery shortcut control 441, the mobile phone can open the gallery application.

[0144] The shutter control 442 can be used to monitor the user operation that triggers the photo. The mobile phone can detect the user operation acting on the shutter control 442. In response to the operation, the mobile phone obtains the corresponding photo image, and synthesizes the final output image and saves it as a picture in the gallery application. In addition, the mobile phone can also display thumbnails of the saved images in the gallery shortcut control 441. In other words, the user can trigger the photo by operating the shutter control 442. It can be understood that the shutter control 442 can be a button or other form of control, and this application does not limit this.

[0145] The camera flip control 443 can be used to monitor the user operation that triggers the camera flip. The mobile phone can detect the user operation acting on the camera flip control 443, such as a touch operation, and in response to the operation, the mobile phone can switch the camera used for shooting, such as switching the rear camera to the front camera, or switching the front camera to the rear camera.

[0146] The shooting interface 400 may also include more or fewer controls, which is not limited in the embodiment of the present application.

[0147] The following is an introduction to the application scenarios of the solutions provided in the embodiments of the present application.

[0148] In a backlit or high-contrast environment, the camera usually enters a variable exposure shooting mode to obtain photos with a high dynamic range. In the prior art, in the variable exposure shooting mode, the camera collects exposure frames with different exposure amounts, that is, a raw frame sequence, and then the processor performs two processing flows before sending the image frame to the display screen.

[0149] The first processing flow is: identify the variable exposure frames from the original frame sequence, discard the variable exposure frames, send the preview frame sequence after discarding the variable exposure frames to the display screen, and the display screen will preview and display it in its preview area after receiving the preview frame sequence. The second processing flow is: perform algorithm fusion processing on the long exposure frames, short exposure frames and normal exposure frames in the original frame sequence to obtain high dynamic range photos.

[0150] See also Figure 5a , if the user opens the mobile phone camera in a backlit scene and uses the variable exposure shooting mode to take a photo of a person running, when the shooting command is generated, 5 frames of images are collected as shown in the figure, among which frame 1, frame 2 and frame 5 are normal exposure frames, frame 3 is a short exposure frame, and frame 4 is a long exposure frame. Figure 5b, because before sending the 5 frames of images to the display screen, the processor will identify the variable exposure frames and discard the variable exposure frames, that is, discard frames 3 and 4. At this time, the preview area of ​​the camera will display frames 1, 2, and 5 one by one. In this way, due to the missing image frames between frames 2 and 5, after the user clicks the shooting control, the preview area of ​​the display screen will pause and the preview screen will jump, resulting in a poor preview experience for the user.

[0151] In view of this, in the embodiments of the present application, the variable exposure photography processing flow in the prior art is improved. In the improved processing flow, when taking variable exposure photos, it is necessary to adjust the image output mode of the camera sensor from the first mode to the second mode, use normal exposure parameters and variable exposure parameters to set the exposure parameters of the camera sensor, obtain the normal exposure frame collected by the camera sensor under the normal exposure parameters, and the variable exposure frame collected under the variable exposure parameters; then send the variable exposure frame to the photography path to generate the photographed film, send the normal exposure frame to the preview path, and preview and display the normal exposure frame. See Figure 5c When the camera sensor's image output mode is switched to the second mode, the camera sensor will output normal exposure frames and variable exposure frames at the same time, and send the normal exposure frames to preview display. Since the exposure of each image frame is consistent when the user previews the image frame, it can be seen that the method of the embodiment of the present application solves the problem of poor preview effect when taking photos with variable exposure.

[0152] The frame preview method provided in the embodiment of the present application is described in detail below through specific examples.

[0153] See also Figure 6 , which is a flowchart of a frame preview method provided in an embodiment of the present application, may include steps S601-S604:

[0154] S601, in response to the image output mode of the camera sensor being adjusted from the first mode to the second mode, setting the camera sensor using sensor exposure parameters, wherein the sensor exposure parameters include normal exposure parameters and variable exposure parameters.

[0155] The function of the camera sensor is to output images in a specific output mode. The first mode can be, for example, a binning mode, a dual conversion gain mode (DCG), a staggered high dynamic range imaging mode (SHDR), etc. The DCG mode is essentially a binning mode.

[0156] The DCG mode means that by adding DCG to the CMOS pixel (pixel), the camera sensor has both high sensitivity and high dynamic range capabilities. DCG includes high gain conversion (High Conversion Gain, HCG) and low gain conversion (Low Conversion Gain, LCG). High HCG conversion gain corresponds to long exposure frames, and the dark area has a good signal-to-noise ratio. Low LCG conversion gain is a short exposure frame, which can ensure good highlight details. When the camera sensor works in DCG mode, long exposure frames and short exposure frames are obtained respectively, and then the long exposure frames and short exposure frames are fused, and the fused image frames are used as the images output by the camera. In other words, the image output by the camera sensor in DCG mode is an image frame after the long exposure frame and the short exposure frame are fused. After the fusion of the two, there is no ghosting problem, and the dynamic range is improved.

[0157] In the embodiment of the present application, after obtaining a shooting request for variable exposure shooting, the image output mode of the camera sensor may be adjusted from the first mode to the second mode. For example, in actual application, the shooting request may be a request triggered by a user clicking a shooting control in a shooting mode in which HDR is turned on in a camera application.

[0158] In response to the image output mode of the camera sensor being adjusted from the first mode to the second mode, the camera sensor is set using normal exposure parameters and variable exposure parameters, that is, the camera sensor captures image frames using normal exposure parameters and variable exposure parameters at the same time.

[0159] Exemplarily, the variable exposure parameter may be each parameter in a preset exposure parameter sequence, and the preset exposure parameter sequence may be set by relevant staff based on experience. After obtaining a photo request for capturing image frames with different exposure parameters (i.e., a photo request for variable exposure photography), the processor obtains each exposure parameter from the preset exposure parameter sequence and sends it to the camera sensor to drive the camera sensor to capture image frames with the different exposure parameters.

[0160] It should be noted that the exposure parameters of each image frame may also be the exposure parameters in the exposure parameter sequence generated in real time by the processor after a photo request is triggered to collect image frames with different exposure parameters. At this time, the processor sends the generated exposure parameter sequence to the camera sensor to drive the camera sensor to collect the image frame sequence with different exposure parameters.

[0161] In an example, the variable exposure parameter may be an exposure parameter whose EV value is not EV0, or an exposure parameter whose EV value is not within a preset range. The preset range may be a brightness range that is basically consistent with the brightness of EV0 and is set by relevant staff based on experience. Both are reasonable.

[0162] S602, obtaining normal exposure frames acquired by the camera sensor under the normal exposure parameters, and variable exposure frames acquired by the camera sensor under the variable exposure parameters.

[0163] In the second mode, the camera sensor simultaneously outputs two frames, one frame is a normal exposure frame acquired based on normal exposure parameters, and the other frame is a variable exposure frame acquired based on variable exposure parameters.

[0164] The "exposure" in the normal exposure frame and the variable exposure frame refers to the exposure value (EV), not the exposure time. The exposure value can be determined by the exposure time and the gain. For example, the exposure value satisfies the formula: EV = Gain * exposure time. Among them, EV represents the exposure value. The factors affecting Gain include but are not limited to CG and Iso. CG is the conversion gain, and Iso is the sensitivity.

[0165] S603, sending the variable exposure frame to a photographing path.

[0166] The acquired variable exposure frames are sent to the photo-taking channel, and a high dynamic range image is obtained by fusing multiple variable exposure frames through a photo-taking algorithm.

[0167] S604: Preview and display the normal exposure frame.

[0168] Since the normal exposure frame is an image frame acquired with normal exposure parameters, the normal exposure frame can be previewed and displayed, and the brightness of the normal exposure frame will not change significantly, and the user will not see that a certain frame is particularly bright or dark during preview.

[0169] In the embodiment of the present application, in response to the camera sensor's image output mode being adjusted from the first mode to the second mode, the camera sensor is set using sensor exposure parameters, wherein the sensor exposure parameters include normal exposure parameters and variable exposure parameters; the normal exposure frame collected by the camera sensor under the normal exposure parameters and the variable exposure frame collected under the variable exposure parameters are obtained; the variable exposure frame is sent to the photo-taking path; and the normal exposure frame is previewed and displayed. This achieves that the image frame can be previewed even in the case of variable exposure photo-taking, and since the previewed frame is a normal exposure frame, the user's preview experience is also improved.

[0170] In an example, when the first mode is the merge mode, a first exposure parameter of the camera sensor in the merge mode is obtained to obtain a normal exposure parameter.

[0171] Before receiving a shooting request for variable exposure photography, the camera sensor outputs an image in a merge mode, and obtains a first exposure parameter of the camera sensor in the merge mode as a normal exposure parameter, wherein the merge mode refers to adding together the charges induced by adjacent pixels and reading them out in a one-pixel mode.

[0172] In an embodiment of the present application, for the case where the first mode is a merge mode, the first exposure parameter of the camera sensor in the merge mode is obtained. In this way, when the camera sensor mode is switched to the second mode, the exposure parameter of the camera sensor can be set with the first exposure parameter and the variable exposure parameter.

[0173] In one example, when the first mode is a dual-gain conversion mode, a short exposure frame acquired by the camera sensor in a low-gain conversion mode and a long exposure frame acquired in a high-gain conversion mode are acquired; wherein the dual-gain conversion mode includes a low-gain conversion mode and a high-gain conversion mode;

[0174] Merging the short exposure frame and the long exposure frame;

[0175] A second exposure parameter of the fused exposure frame is determined to obtain a normal exposure parameter.

[0176] When the first mode is the dual-gain conversion mode, the output mode of the camera sensor is to output two frames at the same time, namely, a short exposure frame corresponding to the low-gain conversion mode and a long exposure frame corresponding to the high-gain conversion mode. At this time, it is necessary to obtain the short exposure frame collected by the camera sensor in the low-gain conversion mode and the long exposure frame collected in the high-gain conversion mode, fuse the short exposure frame and the long exposure frame, and determine the second exposure parameter of the camera sensor in the dual-gain conversion mode based on the fused exposure frame.

[0177] In an embodiment of the present application, for the case where the first mode is a dual gain conversion mode, a short exposure frame collected by the camera sensor in a low gain conversion mode and a long exposure frame collected in a high gain conversion mode are obtained, the short exposure frame and the long exposure frame are fused, and the exposure parameters of the fused exposure frame are determined, that is, the second exposure parameters of the camera sensor in the dual gain conversion mode, to obtain normal exposure parameters. In this way, when the camera sensor mode is switched to the second mode, the exposure parameters of the camera sensor can be set with the second exposure parameters and the variable exposure parameters.

[0178] In one example, when the first mode is a dual gain conversion mode, a third exposure parameter of the camera sensor in a low gain conversion mode and a fourth exposure parameter in a high gain conversion mode are obtained; wherein the dual gain conversion mode includes a low gain conversion mode and a high gain conversion mode, and the normal exposure parameters include a third exposure parameter and a fourth exposure parameter.

[0179] In an embodiment of the present application, when the first mode is the dual gain conversion mode, the image output mode of the camera sensor is to output two frames at the same time, namely, a short exposure frame corresponding to the low gain conversion mode and a long exposure frame corresponding to the high gain conversion mode. At this time, it is necessary to obtain the third exposure parameter of the camera sensor in the low gain conversion mode and the fourth exposure parameter in the high gain conversion mode. In this way, when the camera sensor mode is switched to the second mode, the exposure parameter of the camera sensor can be set with the third exposure parameter, the fourth exposure parameter and the variable exposure parameter.

[0180] In one example, the above step S602, obtaining the normal exposure frame collected by the camera sensor under the normal exposure parameters, includes:

[0181] Acquire a short exposure frame acquired by the camera sensor under a third exposure parameter and a long exposure frame acquired under a fourth exposure parameter.

[0182] After the image output mode of the camera sensor is switched to the second mode, the camera sensor simultaneously collects image frames with the third exposure parameter, the fourth exposure parameter, and the variable exposure parameter. Since the third exposure parameter is an exposure parameter in the low-gain conversion mode, when the camera sensor collects image frames with the third exposure parameter, what is collected is a short-exposure frame; since the fourth exposure parameter is an exposure parameter in the high-gain conversion mode, when the camera sensor collects image frames with the fourth exposure parameter, what is collected is a long-exposure frame.

[0183] The above step S604, previewing and displaying the normal exposure frame, includes:

[0184] Merging the short exposure frame and the long exposure frame;

[0185] The preview shows the blended frame.

[0186] After obtaining the short exposure frame collected by the camera sensor under the third exposure parameter and the long exposure frame collected under the fourth exposure parameter, the short exposure frame and the long exposure frame may be fused and sent for preview display.

[0187] In the embodiment of the present application, for the case where the first mode is a dual gain conversion mode, short exposure frames are captured using the third exposure parameters in the low gain conversion mode, and long exposure frames are captured using the fourth exposure parameters in the high gain conversion mode. The consistency of the preview effect can be ensured by the short exposure frames and the long exposure frames.

[0188] In one example, the method further includes:

[0189] S701, after acquiring a preset number of variable exposure frames, triggering the camera sensor to adjust its image output mode from the second mode to the first mode.

[0190] After the variable exposure photography is completed, that is, after a preset number of variable exposure frames are acquired, which preset number can be determined according to actual conditions, the image output mode of the camera sensor needs to be adjusted to the first mode.

[0191] S702: Restore the exposure parameters of the camera sensor to corresponding exposure parameters in the first mode.

[0192] S703: Drive the camera sensor to capture image frames with corresponding exposure parameters in the first mode.

[0193] In the embodiment of the present application, after the variable exposure photography is completed, the image output mode of the camera sensor is adjusted from the second mode to the first mode, so that the camera sensor still captures image frames with the corresponding exposure parameters in the first mode.

[0194] See also Figure 7 , is a frame preview method in a merging mode provided by an embodiment of the present application. The camera sensor originally works in the merging mode Binning. When receiving a shooting request for variable exposure photography, the image output mode of the camera sensor is triggered to switch from the Binning mode to the staggered high dynamic range imaging mode SHDR. In the SHDR mode, the camera sensor simultaneously outputs variable exposure frames and normal exposure frames, wherein the normal exposure frame is obtained by configuring the exposure parameters and light ratio parameters of the automatic exposure (AE) module unchanged. The variable exposure frame SNSL is sent to the photography path for synthesis of the photographed film, and the normal exposure frame is sent to the preview path for frame preview display. After the variable exposure photography is completed, the image output mode of the camera sensor is triggered to switch from the staggered high dynamic range imaging mode SHDR to the Binning mode. S, N, and L in the figure all represent image frames, S is a short exposure frame, L is a long exposure frame, N is a normal exposure frame, and Nr is a normal exposure frame used as a reference. The embodiment of the present application can realize variable exposure photography preview without frame loss, and can also support the use of a variable exposure photography method when shooting live photos livephoto.

[0195] See also Figure 8, a frame preview method in a dual gain conversion mode provided by an embodiment of the present application, the camera sensor originally works in a dual gain conversion mode DCG, DCG includes high gain conversion (High Conversion Gain, HCG) and low gain conversion (Low Conversion Gain, LCG), the camera sensor collects long exposure frames in HCG mode, and collects short exposure frames in LCG mode, when receiving a shooting request for variable exposure photography, the image output mode of the camera sensor is triggered to switch from DCG mode to staggered high dynamic range imaging mode SHDR, in SHDR mode, the camera sensor simultaneously outputs variable exposure frames and normal exposure frames, wherein the normal exposure frame is obtained by configuring the exposure parameters and light ratio parameters of the automatic exposure (automatic exposure, AE) module unchanged. The variable exposure frame SNSL is sent to the photography path for synthesis of the photographed film; the normal exposure frame is sent to the preview path for frame preview display. After the variable exposure photography is completed, the image output mode of the camera sensor is triggered to switch from the staggered high dynamic range imaging mode SHDR to the DCG mode. In the figure, S, N, and L all represent image frames, S is a short exposure frame, L is a long exposure frame, N is a normal exposure frame, Lr is a long exposure frame used as a reference, and Sr is a short exposure frame used as a reference. The embodiment of the present application can realize variable exposure photo preview without frame loss, and can also support the use of variable exposure photo shooting method when shooting live photos.

[0196] In an example, before triggering variable exposure photography, the image output mode of the camera sensor can also be the interlaced high dynamic range imaging mode SHDR. In this way, when a shooting request for variable exposure photography is received, it is only necessary to configure the exposure parameters through the automatic exposure (AE) module to drive the camera sensor to capture image frames with the configured exposure parameters.

[0197] In an example, in the dual gain conversion mode DCG or staggered high dynamic range imaging mode SHDR, the light ratio is relatively large. If a large light ratio is set for a single frame sent for preview, the signal-to-noise ratio will decrease; however, when previewing the image frame, the resolution of the image frame is low, so the user will not obviously perceive the noise during the preview. Of course, in this case, you can also choose to switch to the SHDR mode that outputs three frames at the same time, two of which are used to keep consistent with HCG and LCG and sent for preview display, thereby ensuring the consistency of the preview effect, and the other frame becomes the exposure frame for taking pictures and processing.

[0198] See also Fig. 9, a frame preview method provided by an embodiment of the present application, the user can click on the camera application, when the user clicks on the camera to take a picture, the photo-taking instruction can be sent to the camera hardware abstraction layer HAL through the camera access interface, HAL can send the photo-taking instruction to the automatic exposure module AE, AE will calculate various exposure parameters, and configure the output mode of the camera sensor to be SHDR mode, as well as the exposure and light ratio parameters of the frame in the SHDR mode. After the camera sensor acquires the image frame, it sends the variable exposure frame to HAL for processing into a picture; at the same time, AE will configure the output mode of the camera sensor to be the original mode, as well as the exposure parameters of the frame in the original output mode, and send the collected frame to HAL.

[0199] On the other hand, the embodiment of the present application provides a frame preview device, see Fig.10 , the device comprises:

[0200] An exposure parameter setting module 801 is used to set the camera sensor using sensor exposure parameters in response to the image output mode of the camera sensor being adjusted from the first mode to the second mode, wherein the sensor exposure parameters include normal exposure parameters and variable exposure parameters;

[0201] An acquisition module 802 is used to acquire a normal exposure frame acquired by the camera sensor under the normal exposure parameter, and a variable exposure frame acquired by the camera sensor under the variable exposure parameter;

[0202] A photographing module 803 is used to send the variable exposure frame to a photographing path;

[0203] The preview module 804 is used to preview and display the normal exposure frame.

[0204] In a possible implementation, the device further includes:

[0205] The first exposure parameter acquisition module is used to acquire the first exposure parameter of the camera sensor in the merge mode to obtain a normal exposure parameter when the first mode is the merge mode.

[0206] In a possible implementation, the device further includes:

[0207] The second exposure parameter acquisition module is used to acquire short exposure frames collected by the camera sensor in a low gain conversion mode and long exposure frames collected in a high gain conversion mode when the first mode is a dual gain conversion mode; wherein the dual gain conversion mode includes a low gain conversion mode and a high gain conversion mode; fuse the short exposure frame and the long exposure frame; determine the second exposure parameter of the fused exposure frame to obtain a normal exposure parameter.

[0208] In a possible implementation, the device further includes:

[0209] The third and fourth exposure parameter acquisition modules are used to obtain the third exposure parameter of the camera sensor in the low gain conversion mode and the fourth exposure parameter in the high gain conversion mode when the first mode is the dual gain conversion mode; wherein the dual gain conversion mode includes the low gain conversion mode and the high gain conversion mode, and the normal exposure parameters include the third exposure parameter and the fourth exposure parameter.

[0210] In a possible implementation manner, the acquisition module is specifically used to:

[0211] Acquire a short exposure frame acquired by the camera sensor under a third exposure parameter and a long exposure frame acquired under a fourth exposure parameter;

[0212] The preview module is specifically used to fuse the short exposure frame and the long exposure frame; and preview and display the fused frame.

[0213] In a possible implementation, the device further includes:

[0214] An image output mode recovery module, used to trigger the image output mode of the camera sensor to be adjusted from the second mode to the first mode after a preset number of variable exposure frames are acquired;

[0215] An exposure parameter recovery module, used to recover the exposure parameters of the camera sensor to the corresponding exposure parameters in the first mode;

[0216] The acquisition module is used to drive the camera sensor to acquire image frames with corresponding exposure parameters in the first mode.

[0217] In a possible implementation, the second mode is an interlaced high dynamic range imaging mode.

[0218] In a possible implementation, the device further includes:

[0219] The image output mode adjustment module is used to trigger the image output mode of the camera sensor to be adjusted from the first mode to the second mode after obtaining a shooting request for variable exposure photography.

[0220] The present application also provides an electronic device, which includes one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute some or all of the steps in the above method embodiments.

[0221] The present application also provides a computer-readable storage medium, including a computer program, which, when executed on an electronic device, enables the electronic device to perform some or all of the steps in the above method embodiment. The above storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0222] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0223] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0224] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0225] In some cases, the disclosed embodiments can be implemented in hardware, firmware, software or any combination thereof. The disclosed embodiments can also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which can be read and executed by one or more processors. For example, instructions can be distributed over a network or through other computer-readable media. Therefore, machine-readable media can include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to, floppy disks, optical disks, optical disks, compact disc read-only memories (Compact Disc Read Only Memory, CD-ROMs), magneto-optical disks, read-only memories, random access memories, erasable programmable read-only memories (Erasable Programmable Read Only Memory, EPROM), electrically erasable programmable read-only memories (Electrically Erasable Programmable Read Only Memory, EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable memories for transmitting information (e.g., carrier waves, infrared signal digital signals, etc.) using the Internet in electrical, optical, acoustic or other forms of propagation signals. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0226] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings of the specification. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0227] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0228] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0229] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A frame preview method, characterized in that: The method comprises: In response to the image output mode of the camera sensor being adjusted from the first mode to the second mode, setting the camera sensor using a sensor exposure parameter, wherein the sensor exposure parameter includes a normal exposure parameter and a variable exposure parameter; Acquire a normal exposure frame acquired by the camera sensor under the normal exposure parameter, and a variable exposure frame acquired by the camera sensor under the variable exposure parameter; Sending the variable exposure frame into a photographing path; The preview displays the normal exposure frame.

2. The method according to claim 1, characterized in that: The method further comprises: When the first mode is the merge mode, a first exposure parameter of the camera sensor in the merge mode is acquired to obtain a normal exposure parameter.

3. The method according to claim 1, characterized in that The method further comprises: When the first mode is a dual-gain conversion mode, a short exposure frame acquired by the camera sensor in a low-gain conversion mode and a long exposure frame acquired in a high-gain conversion mode are acquired; wherein the dual-gain conversion mode includes a low-gain conversion mode and a high-gain conversion mode; Merging the short exposure frame and the long exposure frame; A second exposure parameter of the fused exposure frame is determined to obtain a normal exposure parameter.

4. The method according to claim 1, characterized in that: The method further comprises: When the first mode is the dual gain conversion mode, a third exposure parameter of the camera sensor in the low gain conversion mode and a fourth exposure parameter in the high gain conversion mode are obtained; wherein the dual gain conversion mode includes a low gain conversion mode and a high gain conversion mode, and the normal exposure parameters include a third exposure parameter and a fourth exposure parameter.

5. The method according to claim 4, characterized in that The obtaining of the normal exposure frame collected by the camera sensor under the normal exposure parameters includes: Acquire a short exposure frame acquired by the camera sensor under a third exposure parameter and a long exposure frame acquired under a fourth exposure parameter; The preview displays the normal exposure frame, including: Merging the short exposure frame and the long exposure frame; The preview shows the blended frame.

6. The method according to claim 1, characterized in that The method further comprises: After a preset number of variable exposure frames are acquired, the image output mode of the camera sensor is triggered to be adjusted from the second mode to the first mode; Restoring the exposure parameters of the camera sensor to the corresponding exposure parameters in the first mode; The camera sensor is driven to capture image frames with corresponding exposure parameters in the first mode.

7. The method according to any one of claims 1 to 6, characterized in that: The second mode is an interlaced high dynamic range imaging mode.

8. The method according to claim 1, characterized in that The method further comprises: After obtaining a shooting request for variable exposure photography, the image output mode of the camera sensor is triggered to be adjusted from the first mode to the second mode.

9. A frame preview device, characterized in that: The device comprises: an exposure parameter setting module, configured to, in response to the image output mode of the camera sensor being adjusted from the first mode to the second mode, set the camera sensor using a sensor exposure parameter, wherein the sensor exposure parameter includes a normal exposure parameter and a variable exposure parameter; An acquisition module, used to acquire the normal exposure frame collected by the camera sensor under the normal exposure parameter, and the variable exposure frame collected by the camera sensor under the variable exposure parameter; A photographing module, used for sending the variable exposure frame into a photographing path; The preview module is used to preview and display the normal exposure frame.

10. An electronic device, characterized in that: include: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The method comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the method according to any one of claims 1 to 8.

Images