Photographing method and electronic equipment

Abstract

The embodiment of the invention provides a photographing method and electronic equipment. In the method, the electronic equipment displays an image preview interface of a target application; wherein the target application is an application with a camera function; the preview image frame displayed in the image preview interface is collected based on an image preview request continuously issued by the target application; in response to the received user operation, the target application generates an image shooting request; wherein the exposure parameter of the image shooting request is different from the exposure parameter of the image preview request; and if the image preview request and the image shooting request which are not processed exist at the same time, the electronic equipment calls an image sensor to collect a shot image frame corresponding to the image shooting request. In this way, the image shooting request skips the image preview request to be processed preferentially, so that the image output time of the shot image frame with the exposure parameter changed is shortened.

Description

Photographing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 24, 2022, with application number 202211018122.4 and application name “Photographing method and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of smart terminal technology, and in particular to a photographing method and electronic device. Background Art

[0003] As people's requirements for the functions of electronic devices become increasingly perfect, their demands for the photographic effects of electronic devices are also constantly increasing.

[0004] When a user activates the camera function of an electronic device, a preview image is typically displayed on the electronic device's display. When the electronic device captures a desired image, the user taps the camera control to take the photo. To improve photo quality, the final output image is typically synthesized from image frames captured by the electronic device during the preview mode and image frames generated in response to the user's camera operation. However, shortening the image output time is a challenge that needs to be addressed.

[0005] Summary of the Invention

[0006] To solve the above technical problems, embodiments of the present application provide a photographing method and electronic device, wherein an image capture request that changes image exposure parameters is processed first, skipping image preview requests, thereby shortening the time it takes to output an image frame captured with the changed exposure parameters.

[0007] In a first aspect, an embodiment of the present application provides a photographing method. The method is applied to an electronic device and includes:

[0008] The electronic device displays an image preview interface of a target application; wherein the target application is an application having a camera function; and the preview image frames displayed in the image preview interface are collected based on image preview requests continuously issued by the target application;

[0009] In response to the received user operation, the target application generates an image capture request; wherein the exposure parameter of the image capture request is different from the exposure parameter of the image preview request;

[0010] If there are unprocessed image preview requests and image capture requests at the same time, the electronic device calls the image sensor to collect the captured image frames corresponding to the image capture requests.

[0011] The exposure parameters of the captured image frame and the preview image frame are different.

[0012] During the process of the target application displaying the image preview interface, the target application can send image preview requests at a preset frequency to continuously collect preview image frames corresponding to each image preview request, ensuring that the preview image frames in the image preview interface of the target application are displayed smoothly and smoothly.

[0013] In this way, the image capture request for changing the image exposure parameters will be processed first, skipping the image preview request, thereby shortening the time it takes to output the image frame captured with the changed exposure parameters.

[0014] According to the first aspect, the method further includes: storing image preview requests continuously sent by the target application in a request queue, and storing image capture requests sent by the target application in a target module; wherein the request queue is a first-in-first-out queue.

[0015] Exemplarily, the target module may be the Bridge module mentioned below.

[0016] In this embodiment of the present application, the target module is a new module added to the system's camera path, used to bridge the camera HAL module and the ISP module. The target module is a single instance class used to store image capture requests with different image exposure parameters.

[0017] In this way, unlike the existing technology, the image shooting request sent by the target application is no longer stored in the request queue together with the image preview request, but is stored separately in the target module, so as to avoid the problem that the image shooting request can only be processed after the image preview request that entered the queue earlier than the image shooting request is processed, thereby reducing the path delay of the image shooting request.

[0018] According to the first aspect, or any implementation of the first aspect above, there are unprocessed image preview requests and image capture requests at the same time, which can be specifically as follows: if there is an image preview request in the request queue and an image capture request in the target module, the electronic device determines that there are unprocessed image preview requests and image capture requests at the same time.

[0019] According to the first aspect, or any implementation of the first aspect above, the electronic device includes a camera hardware abstraction layer HAL module, an image signal processing ISP module, and an image sensor driver module, and the method includes:

[0020] The camera HAL module obtains an image preview request in the request queue in turn and sends the image preview request to the ISP module;

[0021] The ISP module receives the image preview request;

[0022] If the ISP module reads an image capture request in the target module, it sends the image capture request to the image sensor driver module, so that the image sensor driver module drives the image sensor to capture an image frame corresponding to the image capture request;

[0023] If the ISP module does not read the image capture request in the target module, it sends the image preview request to the image sensor driver module, so that the image sensor driver module drives the image sensor to capture the preview image frame corresponding to the image preview request.

[0024] In this way, before sending an image preview request to the image sensor driver module, the ISP module will first determine whether there is an image capture request. If so, the image capture request will be processed first, otherwise the image preview request will be processed. This ensures that the image capture request can be processed in a timely manner and reduces the time it takes to output the captured image frame.

[0025] According to the first aspect, or any implementation of the first aspect above, the method further includes: if the ISP module reads an image capture request in the target module, discarding the image preview request.

[0026] In this way, when there are both unprocessed image preview requests and image capture requests, the image capture request replaces the image preview request and is processed. This reduces the time it takes to output the captured image frame by putting the image capture request in the queue. Furthermore, discarding the replaced image preview request will not affect subsequent image preview requests.

[0027] According to the first aspect, or any implementation of the first aspect above, the method further includes: the camera HAL module receives an image preview request or an image capture request issued by the target application, and stores the image preview request in a request queue, and stores the image capture request in the target module.

[0028] According to the first aspect, or any implementation of the first aspect above, the method further includes: the ISP module sends the preview image frame or the captured image frame fed back by the image sensor driver module to the camera HAL module; the camera HAL module stores the preview image frame or the captured image frame in a zero-second delay queue in turn.

[0029] According to the first aspect, or any implementation of the first aspect above, the method further includes: in a multi-frame synthetic shooting scenario, the camera HAL module reads a corresponding number of preview image frames and shot image frames in a zero-second delay queue, and uses an image processing algorithm that matches the shooting mode to perform a multi-frame image synthesis operation to obtain a target image; wherein the target image is used to be stored in a gallery application.

[0030] Thus, in a multi-frame fusion photography scenario, since the image capture request skips the individual image preview requests in the request queue, the frame interval between the captured image frame corresponding to the image capture request and the preview image frame in the ZSL queue is smaller, thereby improving the continuity of the multiple frames used to synthesize the final output image and enhancing the image quality of the multi-frame synthesis. In particular, in dynamic photography scenarios, the image frames captured by the image sensor may exhibit "smearing." Therefore, the smaller the frame interval between the captured image frame and the preview image frame, the smaller the image difference between the captured image frame and the preview image frame, resulting in a better final image synthesized from multiple frames, thus avoiding the problem of motion blur in the image.

[0031] According to the first aspect, or any implementation of the first aspect above, the target application is a camera application.

[0032] In a second aspect, embodiments of the present application provide an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the photographing method of the first aspect and any one of the first aspects.

[0033] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0034] In a third aspect, embodiments of the present application provide a computer-readable storage medium comprising a computer program, which, when executed on an electronic device, causes the electronic device to execute the photographing method of the first aspect and any one of the first aspects.

[0035] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0036] In a fourth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed, enables a computer to execute the photographing method according to the first aspect or any one of the first aspects.

[0037] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0038] In a fifth aspect, the present application provides a chip comprising a processing circuit and a transceiver pin. The transceiver pin and the processing circuit communicate with each other via an internal connection path, and the processing circuit executes the photographing method according to the first aspect or any one of the first aspects to control the receiving pin to receive a signal and to control the transmitting pin to send a signal.

[0039] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] FIG1 is a schematic diagram showing a hardware structure of an electronic device;

[0041] FIG2 is a schematic diagram illustrating a software structure of an electronic device;

[0042] FIG3 a is an exemplary diagram of one of the user interfaces of an electronic device;

[0043] FIG3 b is an exemplary diagram of one of the user interfaces of an electronic device;

[0044] FIG4a is an exemplary diagram of a processing flow of an existing image preview request;

[0045] FIG4 b is an exemplary diagram showing a processing flow of an existing image capture request;

[0046] FIG4c is an exemplary diagram showing an output sequence of an existing preview image frame and a captured image frame;

[0047] FIG5 is a schematic diagram of a photographing process according to an embodiment of the present application;

[0048] FIG6 a is a schematic diagram of a storage process of an image preview request and an image capture request provided in an embodiment of the present application;

[0049] FIG6 b is an example of a processing flow of an image preview request and an image capture request provided in an embodiment of the present application;

[0050] FIG6 c is an example of a processing flow of an image preview request and an image capture request provided in an embodiment of the present application;

[0051] FIG6 d is an example of a processing flow of an image preview request and an image capture request provided in an embodiment of the present application;

[0052] FIG7 is a schematic diagram of a photographing process according to an embodiment of the present application;

[0053] FIG8a is an example of multi-frame image synthesis provided by an embodiment of the present application;

[0054] FIG8 b is an example of multi-frame image synthesis provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0056] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0057] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0058] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0059] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0060] The photographing method provided in the embodiments of the present application can be applied to electronic devices. Optionally, the electronic devices in the embodiments of the present application can be mobile phones with photographing functions, sports cameras (GoPro), digital cameras, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, vehicle-mounted devices, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, etc. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic devices.

[0061] As shown in Figure 1, a schematic diagram of the structure of an electronic device 100 is shown. Optionally, the electronic device 100 can be a terminal, which can also be called a terminal device. The terminal can be a device with a camera, such as a cellular phone (such as a mobile phone) or a tablet computer (pad), which is not limited in this application. It should be understood that the electronic device 100 shown in Figure 1 is only an example of an electronic device, and the electronic device 100 can have more or fewer components than shown in the figure, can combine two or more components, or can have different component configurations. The various components shown in Figure 1 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0062] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, an air pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0063] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0064] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0065] The processor 110 may further include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0066] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0067] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0068] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0069] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0070] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0071] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0072] The wireless communication module 160 can provide wireless communication solutions for application on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0073] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0074] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0075] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0076] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0077] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0078] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0079] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

[0080] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, such as enabling the electronic device 100 to implement the photo-taking method in the embodiment of the present application. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0081] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0082] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0083] Speaker 170A, also known as a "horn," is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music or make hands-free calls through speaker 170A. In some embodiments, electronic device 100 can be provided with multiple speakers 170A.

[0084] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0085] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0086] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0087] The pressure sensor is used to sense pressure signals and convert the pressure signals into electrical signals. In some embodiments, the pressure sensor can be provided on the display screen 194. The electronic device 100 can also calculate the touch position based on the detection signal of the pressure sensor.

[0088] The gyroscope sensor can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y, and z axes) can be determined by the gyroscope sensor.

[0089] The accelerometer can detect the magnitude of acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the accelerometer can detect the magnitude and direction of gravity. The accelerometer can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0090] A touch sensor, also known as a "touch panel," can be mounted on display screen 194. The touch sensor and display screen 194 together form a touch screen, also known as a "touch screen." The touch sensor detects touch operations applied to or near the touch sensor. The touch sensor can communicate the detected touch operations to the application processor to determine the type of touch event.

[0091] Keys 190 include a power button (also known as a power button), a volume button, and the like. Keys 190 may be mechanical keys or touch-sensitive keys. Electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of electronic device 100.

[0092] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or touch vibration feedback. For example, touch operations on different applications (such as taking photos, playing audio, etc.) can correspond to different vibration feedback effects.

[0093] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0094] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. It should be noted that in the embodiments of the present application, the operating system of the electronic device may include but is not limited to Symbian, Android, Windows, iOS, Blackberry, Harmony and other operating systems, and this application does not limit them.

[0095] The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100.

[0096] FIG2 is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application.

[0097] The layered architecture of electronic device 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided into four layers: from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, the hardware abstraction layer (HAL), and the kernel layer.

[0098] The application layer can include a series of application packages.

[0099] As shown in Figure 2, the application package may include a camera, a gallery, and third-party applications with camera functions, etc. Exemplarily, the application package may also include applications such as calls, calendars, maps, navigation, music, videos, and short messages.

[0100] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0101] As shown in FIG2 , the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, a camera service, and the like.

[0102] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0103] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0104] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0105] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0106] The Notification Manager allows applications to display notifications in the status bar. These messages can be used to convey informational messages and can disappear automatically after a short pause, without requiring user interaction. For example, notifications are used to notify the completion of downloads, message reminders, and so on. Notifications can also appear in the system's top status bar as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Notifications can take the form of text messages in the status bar, beeping sounds, vibrating electronic devices, or flashing indicator lights.

[0107] The camera service is used to call the camera (including the front camera and / or the rear camera) in response to the application's request. Specifically in this embodiment, the camera service can respond to the image preview request and image capture request issued by the application, and then call the camera to perform image acquisition according to the corresponding request to obtain the corresponding image frame.

[0108] Android Runtime includes core libraries and a virtual machine. Android Runtime is responsible for scheduling and management of the Android system.

[0109] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0110] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0111] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0112] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0113] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0114] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0115] A 2D graphics engine is a drawing engine for 2D drawings.

[0116] The HAL is the interface layer between the operating system kernel and the hardware circuitry. The HAL layer includes, but is not limited to, a camera HAL module and an audio HAL module. The camera HAL module processes image streams, while the audio HAL module processes audio streams (for example, performing noise reduction and directional enhancement).

[0117] The kernel layer is the layer between hardware and software. It includes at least the display driver, audio driver, image sensor driver module, and image signal processing module. The hardware includes at least the processor, display, camera, and image signal processing (ISP). The image signal processing module processes image preview and capture requests based on instructions from the camera HAL module, enabling the image sensor driver module to capture images according to the corresponding requests. The image signal processing module can also process the raw images captured by the camera, such as performing image denoising and image optimization.

[0118] It is understood that the layers in the software structure shown in FIG2 and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, and this application does not limit this.

[0119] It is understandable that in order to implement the photographing method in the embodiment of the present application, the electronic device includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0120] First, some of the terms and related technologies involved in this application are explained to facilitate understanding by those skilled in the art.

[0121] Exposure is the fundamental process and key characteristic of all photochemical imaging methods. Visible images are obtained through exposure. Exposure modes, which refer to the computer's use of natural light sources, are generally classified into various modes, including manual and automatic. The quality of a photograph is closely related to exposure, specifically the amount of light that should pass through to the photosensitive element to produce a clear image. Exposure is determined by exposure time and aperture size.

[0122] Sensitivity, also known as ISO (International Organization for Standardization) value, is a measure of a film's sensitivity to light. It is determined by sensitometry, a technique involving the measurement of several values, and has recently been standardized by the International Organization for Standardization. Less sensitive films require longer exposure times to achieve the same image quality as more sensitive films, and are therefore often referred to as slow films. Highly sensitive films are therefore called fast films. In both digital and film photography, using higher sensitivities to reduce exposure times generally results in reduced image quality (due to coarser film grain, increased image noise, or other factors).

[0123] Exposure is a crucial aspect of photography, determining how much light the photosensitive element captures when capturing an image. In other words, exposure determines the brightness (i.e., light and dark) of the image. Exposure value, on the other hand, reflects the amount of exposure and is a parameter that specifically characterizes how much light the photosensitive element captures. The definition of exposure value shows that exposure value is determined by exposure time, the area through which light passes, and the intensity of ambient light. Shutter speed determines exposure time, while aperture size determines the area through which light passes. Generally speaking, when shooting under natural light, it's nearly impossible to change the ambient light. However, ISO can affect the sensitivity of the photosensitive element to light, so it can be considered that ISO affects the intensity of ambient light captured by the photosensitive element.

[0124] In summary, exposure is affected by three factors: exposure time (shutter speed), aperture size, and ISO. These three factors can be referred to as exposure parameters. EV (Exposure Value) is a numerical value representing the light transmission capacity of a camera lens, derived from the combination of shutter speed and aperture. The slower the shutter speed and the larger the aperture, the longer the light transmission time and area, resulting in a greater exposure.

[0125] The following describes a photo-taking scenario provided by an embodiment of the present application. In this scenario, a mobile phone is used as an example for explanation.

[0126] It is understood that the term "user interface" in the specification, claims and drawings of this application refers to the media interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0127] Figure 3a exemplarily shows an exemplary user interface 300 for displaying applications on a mobile phone. The user interface 300 displays a page with application icons, which may include multiple application icons (for example, a clock application icon, a calendar application icon, a gallery application icon, a memo application icon, an email application icon, an application store application icon, a settings application icon, etc.). A page indicator may also be displayed below the above-mentioned multiple application icons to indicate the positional relationship between the currently displayed page and other pages. There are multiple tray icons (for example, a camera application icon 310, a contact application icon, a phone application icon, and an information application icon) below the page indicator. The tray icon remains displayed when the page is switched. The embodiment of the present application does not limit the content displayed on the user interface 300.

[0128] It is understandable that the mobile phone can detect the user operation (such as touch / click operation) of the user on the camera application icon 310, and in response to the operation, the mobile phone can display the shooting interface 400 shown in Figure 3b. The shooting interface 400 can be the user interface of the default shooting mode of the camera application, and the user can complete the shooting on this interface. The camera application is an image shooting application on electronic devices such as smartphones and tablets. This application does not limit the name of this application. In other words, the user can open the shooting interface 400 of the camera application by clicking the camera application icon 310. It is understandable that the default camera in the default shooting mode is not limited to the rear camera. The mobile phone can also set the front camera as the default camera. That is, after opening the camera application, the mobile phone can display the image captured by the front camera in the preview area 420, which can be used for the user to take pictures with the default front camera.

[0129] Figure 3b illustrates a user interface 400 for a mobile phone camera application. As shown in Figure 3b, shooting interface 400 may include a parameter adjustment area 410, a preview area 420, a camera mode selection area 430, a gallery shortcut control 441, a shutter control 442, and a camera flip control 443. The various controls in parameter adjustment area 410 are used to adjust corresponding shooting parameters, including but not limited to: flash settings, AI recognition on / off settings, color standard settings, and more detailed camera settings.

[0130] The preview area 420 can be used to display a preview image, which is an image collected by the mobile phone through the camera in real time. The mobile phone can refresh the display content in the preview area 420 in real time so that the user can preview the image currently collected by the camera.

[0131] One or more shooting mode options may be displayed in the camera mode option 430. These one or more shooting mode options may include: aperture mode option 431, night mode option 432, portrait mode option 433, photo mode option 434, video mode option 435, professional option mode 436, and more option 437. It is understood that these one or more shooting mode options may be displayed as text information on the interface, such as "aperture", "night", "portrait", "photo", "video", "professional", and "more", or as icons or other forms of interactive elements (IEs), which are not limited in this application.

[0132] When a user operation on a shooting mode option is detected, the mobile phone can activate the shooting mode selected by the user. In particular, when a user operation on more options 437 is detected, the mobile phone can further display more other shooting mode options, such as a time-lapse photography mode option, a continuous shooting mode option, etc., to present the user with richer camera functions. It is understandable that the camera mode options 430 can also include more or fewer shooting mode options. The camera mode options shown in Figure 3b are only one implementation of the present application and should not be regarded as limiting the present application.

[0133] The gallery shortcut control 441 can be used to open the gallery application. In response to a user operation on the gallery shortcut control 441, such as a touch operation, the mobile phone can open the gallery application.

[0134] The shutter control 442 can be used to monitor user operations that trigger taking photos. The mobile phone can detect user operations acting on the shutter control 442. In response to the operation, the mobile phone obtains the corresponding image frame and synthesizes the final output image to save it as a picture in the gallery application. In addition, the mobile phone can also display thumbnails of the saved images in the gallery shortcut control 441. In other words, the user can trigger taking photos by operating the shutter control 442. It is understandable that the shutter control 442 can be a button or other form of control, and this application does not limit this.

[0135] Camera flip control 443 can be used to monitor user operations that trigger camera flipping. The phone can detect user operations on camera flip control 443, such as touch operations, and in response to the operations, the phone can switch the camera used for shooting, such as switching the rear camera to the front camera, or switching the front camera to the rear camera.

[0136] The shooting interface 400 may also include more or fewer controls, which is not limited in the embodiment of the present application.

[0137] Continuing with reference to Figure 3b, the user clicks the shutter control 442. In response to the user operation, the camera application sends a photo request to the kernel layer. After receiving the photo request, the kernel layer reports the subsequent continuous multiple frames of image data output by the image sensor to the camera HAL module. The camera HAL module will call back a photo frame and thumbnail to the camera application. The camera application can confirm that the photo is completed based on the received photo frame or thumbnail, and then restore the shutter control 442 to the enabled state, so that the user can take the next photo.

[0138] Currently, to improve photo quality, the final output image is typically synthesized based on preview image frames captured by the electronic device in the preview state and photo image frames obtained in response to the user's photo operation. Preview image frames are captured based on image preview requests issued by the camera application, and photo image frames are captured based on image capture requests issued by the camera application. It should be noted that to improve the quality of the final output image, the image preview request and the image capture request carry different exposure parameters, including but not limited to EV value and ISO value.

[0139] For example, when the ambient light is low, the exposure parameters carried in the image capture request generated by the camera application in response to the user's operation will be stronger than the exposure parameters carried in the image preview request, so that the brightness of the image frame captured by the image sensor based on the image capture request will be higher. For another example, when the ambient light is bright, the exposure parameters carried in the image capture request generated by the camera application in response to the user's operation will be weaker than the exposure parameters carried in the image preview request, so as to prevent the image frame captured by the image sensor based on the image capture request from being overexposed, resulting in unclear image details. Therefore, compared with the image preview request, the image capture request can be called a capture request that changes the image exposure.

[0140] Continuing with Figure 3b, while the preview image is displayed in preview area 420, the camera application sends an image preview request to the kernel layer at a preset frequency (e.g., dozens of times per second). After receiving the image preview request, the kernel layer drives the image sensor to capture the preview image frame corresponding to the image preview request. Similarly, when the user clicks the shutter control, in response to the user operation, the camera application sends an image capture request to the kernel layer. After receiving the image capture request, the kernel layer drives the image sensor to capture the captured image frame corresponding to the image capture request. The captured image frame can be understood as an image frame obtained by changing the exposure parameters.

[0141] It should be noted that, in response to user operations, the camera application may send one or more image capture requests to the kernel layer. Depending on the shooting mode, the number of image capture requests sent by the camera application to the kernel layer may not be the same; for example, two or four image capture requests may be sent consecutively. The timing of two adjacent image capture requests may be the same or different, and the exposure parameters of the image capture requests sent by the camera application to the kernel layer may be the same or different, which is not limited in the present embodiment.

[0142] The following describes in detail the processing flow of image preview request and image capture request in the multi-frame image synthesis shooting scenario with reference to FIG. 4 a to FIG. 4 c .

[0143] 4a , illustratively, when a camera application displays a preview image in preview area 420, the camera application generates an image preview request 57 and sends the image preview request 57 to the camera HAL module in the HAL via the camera server. The image preview request 57 includes, but is not limited to, the request type, exposure parameters (such as EV value, ISO value, etc.), resolution, and jitter amount.

[0144] After receiving the image preview request 57, the camera HAL module stores it in a request queue. This request queue sequentially stores requests from the camera application, including but not limited to image preview requests and image capture requests. This request queue is a first-in, first-out queue. The camera HAL module sequentially sends each request in the request queue to the ISP module in the kernel layer for processing.

[0145] Continuing with FIG. 4a , illustratively, the camera HAL module receives an image preview request 52 from the request queue and sends it to the ISP module. Upon receiving the image preview request 52, the ISP module sends it to the image sensor driver module, causing the image sensor driver module to drive the image sensor to capture image data according to image acquisition parameters matching the image preview request 52. Furthermore, the image sensor driver module can send a preview image frame 52 generated by the image sensor that matches the image preview request 52 to the ISP module, which then transmits the preview image frame 52 to the camera HAL module. The ISP module can perform raw image data processing on the preview image frame, such as image denoising and image optimization.

[0146] After the camera HAL module receives the preview image frame 52, it stores the preview image frame 52 in the ZSL (Zero Shutter Lag) queue. Among them, the electronic device caches the preview image and the actual captured image in a chronological order in an image queue, which is called a ZSL queue. It should be noted that the images in the ZSL queue are usually RAW images. Among them, the significance of setting the ZSL queue is to eliminate the delay when taking pictures with a mobile phone camera and provide a "shoot and see" experience.

[0147] It should be noted that the image frames stored in the ZSL queue are all newly generated. The embodiment of the present application does not limit the number of image frames stored in the ZSL queue.

[0148] While displaying the preview image in the camera application preview area 420, the camera HAL module continuously reads the preview image from the ZSL queue and sends it to the camera application for display via the Camera Server. For example, referring to FIG. 4a , the camera HAL module reads the preview image 51 from the ZSL queue and sends it to the camera application for display via the Camera Server.

[0149] It should be pointed out that in the above process, the operation of the camera HAL module storing the received image preview request in the request queue, the operation of the camera HAL module reading the image preview request in the request queue and sending it to the ISP module, and the operation of the camera HAL module reading the preview image frame in the ZSL queue and sending it to the camera application via the camera service are performed asynchronously.

[0150] In the scenario shown in FIG4a , for example, preview image frame 50 is currently being displayed, preview image frame 51 has just been generated and is ready for display, and preview image frame 52 is currently being generated. Simultaneously, multiple image preview requests are currently in the request queue, awaiting the generation of preview image frames. In the scenario shown in FIG4a , the image preview requests and the sequence numbers of the preview image frames are merely exemplary and are not intended to be limiting in this embodiment of the present application.

[0151] For example, assume that while the camera application is displaying preview image frame 50, preview image frame 51 has just been generated and is ready for display, and preview image frame 52 is being generated, the user clicks the shutter control to trigger the camera application's capture process. In response to the user's operation, the camera application sends an image capture request to the camera HAL module via the camera service. The exposure parameters corresponding to the image capture request are different from the exposure parameters corresponding to the image preview request.

[0152] Referring to FIG4B , in response to a user clicking the shutter control, the camera application generates an image capture request 58 and sends the image capture request 58 to the camera HAL module in the HAL via the camera service. The image capture request 58 includes, but is not limited to, the request type, exposure parameters (such as EV value, ISO value, etc.), resolution, and shake amount. The exposure parameters corresponding to the image capture request 58 are different from the exposure parameters corresponding to the image preview request 57.

[0153] After receiving the image capture request 58, the camera HAL module stores the image capture request 58 in the request queue. For example, referring to FIG4b, there are image preview requests 56 and 57 in the request queue that have not been processed by the camera HAL module.

[0154] Continuing with FIG4b , illustratively, the camera HAL module receives an image preview request 55 from the request queue and sends it to the ISP module. Upon receiving the image preview request 55, the ISP module sends it to the image sensor driver module, causing the image sensor driver module to drive the image sensor to capture image data according to image acquisition parameters matching the image preview request 55. Furthermore, the image sensor driver module can send a preview image frame 55 generated by the image sensor that matches the image preview request 55 to the ISP module, which then transmits the preview image frame 55 to the camera HAL module. The ISP module can perform raw image data processing on the preview image frame, such as image denoising and image optimization.

[0155] After receiving the preview image frame 55 , the camera HAL module stores the preview image frame 55 in the ZSL queue, waiting for the camera HAL module to take it out of the ZSL queue and send it to the camera application for display via the camera service.

[0156] While displaying the preview image in the camera application preview area 420, the camera HAL module continuously reads the preview image from the ZSL queue and sends it to the camera application for display via the Camera Server. For example, referring to FIG. 4b , the camera HAL module reads the preview image 54 from the ZSL queue and sends it to the camera application for display via the Camera Server.

[0157] Similarly, in the above process, the operation of the camera HAL module storing the received image capture request in the request queue, the operation of the camera HAL module reading the image preview request in the request queue and sending it to the ISP module, and the operation of the camera HAL module reading the preview image frame in the ZSL queue and sending it to the camera application via the camera service are performed asynchronously.

[0158] If the camera application generates multiple image capture requests in response to a user clicking the shutter control, the camera application must sequentially send each of these requests to the camera HAL module for storage in a request queue for processing. For example, assume that in response to a user clicking the shutter control, the camera application generates two image capture requests, such as image capture request 58 and image capture request 59. The process for issuing image capture request 59 can be found in the process for issuing image capture request 58, and will not be further described in this embodiment.

[0159] 4 b , after the camera HAL module reads the image preview request 57 in the request queue, it will continue to read the image capture request 58 in the request queue to capture the captured image frame 58 corresponding to the image capture request 58 .

[0160] When the shutter control is restored to the enabled state, the camera application continues to generate image preview requests at a preset frequency, and sends the image preview requests to the camera HAL module through the camera service, so that the camera application preview area 420 can continue to display the preview image.

[0161] As shown in Figure 4c, the order in which image preview requests and image capture requests enter the request queue is, for example, image preview request 56, image preview request 57, image capture request 58, image capture request 59, image preview request 60, image preview request 61, and image preview request 62. The camera HAL module sequentially obtains and processes each image preview request or image capture request in the order in which the requests exit the queue, thereby capturing the image frame corresponding to the corresponding image preview request or image capture request through the image sensor. Therefore, the order in which image frames are generated is: preview image frame 56, preview image frame 57, capture image frame 58, capture image frame 59, preview image frame 60, preview image frame 61, and preview image frame 62.

[0162] In multi-frame composite capture scenarios, to achieve the "shoot and see" effect, preview image frames are read from the ZSL queue. Only the exposure parameters need to be changed to recapture the image capture frames (e.g., capture frame 58 and capture frame 59). However, each image capture request is processed only after all image preview requests queued before the image capture request are processed. This undoubtedly affects the generation of the image capture frames with changed exposure and causes a delay in processing the image capture request.

[0163] To address the above issues, embodiments of the present application provide a photography method. In a photography scenario where exposure parameters are changed, an image capture request (whose exposure parameters differ from those of an image preview request) is sent directly to the ISP module in a queue-jumping manner, skipping over all image preview requests in the request queue in the HAL. The request is then sent directly to the ISP module to capture the image through the image sensor, thereby shortening the time it takes to output the captured image frame.

[0164] FIG5 is a schematic diagram showing the interaction between the modules. Referring to FIG5 , the process of the photographing method provided by the embodiment of the present application specifically includes:

[0165] S601: The camera application sends an image acquisition request to the camera service.

[0166] The image acquisition request may be an image preview request or an image capture request.

[0167] Exemplarily, in response to a user clicking on the camera application icon, the camera application starts and sends multiple image preview requests to the camera service. While the camera application is displaying preview images, the camera application continues to send image preview requests to the camera service at a preset frequency.

[0168] For example, in response to the user clicking the shutter control, the camera application sends an image capture request to the camera service. In different shooting modes, the number of image capture requests sent by the camera application can be one or more, which is not limited in this embodiment.

[0169] The image acquisition request may include but is not limited to the request type, exposure parameters (such as EV value, ISO value, etc.), resolution, jitter amount, etc.

[0170] In order to improve the quality of the final output image, the exposure parameters carried in the image preview request and the image capture request are different, and the exposure parameters include but are not limited to EV value and ISO value. For example, when the ambient light is dark, the exposure parameters carried in the image capture request generated by the camera application in response to the user's operation will be stronger than the exposure parameters carried in the image preview request, so that the brightness of the image frame captured by the image sensor based on the image capture request will be higher. For another example, when the ambient light is bright, the exposure parameters carried in the image capture request generated by the camera application in response to the user's operation will be weaker than the exposure parameters carried in the image preview request, so as to prevent the image frame captured by the image sensor based on the image capture request from being overexposed, resulting in unclear image details. Therefore, compared with the image preview request, the image capture request can be called a capture request that changes the image exposure.

[0171] S602: The camera service sends an image acquisition request to the camera HAL module.

[0172] The camera service receives the image acquisition request, performs the corresponding processing, such as creating a corresponding service instance, and sends the image acquisition request to the camera HAL module. The relevant processing of the camera service can be referred to in the existing technology and will not be repeated here.

[0173] S603: The camera HAL module receives an image acquisition request. If the image acquisition request is an image preview request, the image preview request is stored in a request queue; if the image acquisition request is an image capture request, the image capture request is stored in a Bridge module.

[0174] Exemplarily, after receiving the image acquisition request, the camera HAL module identifies whether the image acquisition request is an image preview request or an image capture request according to the request type carried in the image acquisition request.

[0175] If the image preview request is received, the camera HAL module stores the image preview request in a request queue, which is a first-in-first-out queue.

[0176] If the received image capture request is an image capture request, the camera HAL module stores the image capture request in the Bridge module. The Bridge module is only an exemplary name and can also be called a priority module, a queue-jumping module, etc.

[0177] In the embodiments of the present application, the Bridge module is a new module added to the system's camera path, used to bridge the camera HAL module and the ISP module. The Bridge module is a singleton class that stores image capture requests (different from the exposure parameters of image preview requests). For example, the Bridge module can also be implemented using a first-in-first-out queue.

[0178] Therefore, different from the existing technology, the camera HAL stores the received image preview request and image capture request separately instead of storing them both in the existing request queue.

[0179] As shown in FIG6 a , illustratively, after receiving the image preview request 57 , the camera HAL stores the image preview request 57 in a first-in-first-out request queue.

[0180] Continuing with FIG6a , illustratively, after the camera HAL receives image capture request 58, it stores image capture request 58 in the Bridge module. Similarly, after the camera HAL receives image capture request 59, it stores image capture request 59 in the Bridge module. The image capture requests in the Bridge module are also stored in the order in which they were received.

[0181] S604: The camera HAL module reads the image preview request from the request queue and sends the image preview request to the ISP module.

[0182] The camera HAL module reads the image preview request in the request queue and sends instructions and the corresponding image preview request to the ISP to perform corresponding operations through the ISP module.

[0183] The camera HAL module receives and stores the request sent by the camera application, and the camera HAL module reads the image preview request in the request queue and sends it to the ISP module. This is an asynchronous process. This embodiment does not limit the execution order of S603 and S604.

[0184] S605: After receiving the image preview request, the ISP module reads whether there is an image capture request in the Bridge module.

[0185] After receiving the image preview request from the camera HAL module, the ISP module writes it into the corresponding register. Before sending the image preview request to the image sensor driver module, the ISP module first checks whether there is an image capture request in the Bridge module.

[0186] S606: If there is no image capture request in the Bridge module, the ISP module sends an image preview request to the image sensor driver module.

[0187] If the ISP module reads that there is no image capture request in the Bridge module, it directly sends the image preview request written in the register to the image sensor driver module, so that the image sensor can perform image acquisition operations according to the corresponding image acquisition parameters.

[0188] S607: If there is an image capture request in the Bridge module, the ISP module reads one image capture request in sequence and sends the image capture request to the image sensor driving module.

[0189] If the ISP module reads an image capture request from the Bridge module, it will read an image capture request from the Bridge module, write it into the register, and send the image capture request to the image sensor driver module so that the image sensor can perform image capture operations according to the corresponding image capture parameters.

[0190] Among them, when there is only one image shooting request in the Bridge module, the ISP module reads the image shooting request and sends it to the image sensor driver module; when there are multiple image shooting requests in the Bridge module, the ISP module will read an image shooting request in the order of storage and send it to the image sensor driver module.

[0191] S608: If there is an image capture request in the Bridge module, the ISP module discards the image preview request.

[0192] Correspondingly, if the ISP module reads that there is an image capture request in the Bridge module, it will discard the received image preview request, that is, discard the image preview request written into the register.

[0193] In this way, when there is an image shooting request in the Bridge module, the image preview request that the ISP module originally wanted to send to the image sensor driver module will be replaced by the image shooting request, and the corresponding image shooting request will be sent to the image sensor driver module for priority processing.

[0194] S609 , the image sensor driving module collects image frames according to the shooting parameters corresponding to the image preview request or the image shooting request, and sends the image frames to the ISP module.

[0195] S610: The ISP module sends the image frame to the camera HAL module.

[0196] S611: The camera HAL module stores the image frame in the ZSL queue.

[0197] In a specific example, in response to a user clicking a shutter control, the camera application sends two image capture requests (assuming they are image capture request 58 and image capture request 59). The exposure parameters of the image capture requests are different from the exposure parameters of the image preview request sent previously.

[0198] As shown in Figure 6b, the camera HAL module illustratively reads an image preview request 56 from the request queue and sends it to the ISP module. Before sending image preview request 56 to the image sensor driver module, the ISP module reads an image capture request 58 from the Bridge module. The ISP module then sends image capture request 58 to the image sensor driver module and discards image preview request 56. At this point, the image sensor driver module performs an image capture operation based on the image capture parameters corresponding to image capture request 58, obtaining a captured image frame 58 corresponding to image capture request 58. This shortens the time it takes to output captured image frame 58.

[0199] Continuing with Figure 6c, illustratively, the camera HAL module reads image preview request 57 from the request queue and sends it to the ISP module. Before sending image preview request 57 to the image sensor driver module, the ISP module reads image capture request 59 from the Bridge module. The ISP module then sends image capture request 59 to the image sensor driver module and discards image preview request 57. At this point, the image sensor driver module performs an image capture operation based on the image capture parameters corresponding to image capture request 59, obtaining a captured image frame 59 corresponding to image capture request 59. This shortens the time it takes to output captured image frame 59.

[0200] Continuing with Figure 6d, the camera HAL module illustratively reads an image preview request 60 from the request queue and sends it to the ISP module. Before sending the image preview request 60 to the image sensor driver module, the ISP module, if it has not read an image capture request from the Bridge module, sends the image preview request 60 to the image sensor driver module. The image sensor driver module then performs an image capture operation based on the image capture parameters corresponding to the image preview request 60, obtaining a preview image frame 60 corresponding to the image preview request 60.

[0201] In this way, in order to optimize photography, in the photography scenario where the exposure parameters are changed, the image preview request is still stored in the request queue and processed in sequence. The image capture request (whose exposure parameters are different from the image preview request) is no longer stored in the request queue and processed in the order of request. Instead, the HAL skips the various image preview requests in the request queue and directly sends them to the ISP module in a queue-jumping manner to capture images through the image sensor, thereby shortening the time it takes to output the captured image frames.

[0202] In multi-frame fusion photography scenarios, since the image capture request skips the individual image preview requests in the request queue, the frame interval between the captured image frame corresponding to the image capture request and the preview image frame in the ZSL queue is smaller, thereby improving the continuity of the multiple frames used to synthesize the final output image and enhancing the image quality of the multi-frame synthesis. In particular, in dynamic photography scenarios, the image frames captured by the image sensor may have "smears." Therefore, the smaller the frame interval between the captured image frame and the preview image frame, the smaller the image difference between the captured image frame and the preview image frame, resulting in a better final image synthesized from multiple frames, thus avoiding the problem of motion blur in the image.

[0203] FIG7 is a schematic diagram of the interaction between the modules. Referring to FIG7 , the process of the photographing method provided by the embodiment of the present application specifically includes:

[0204] S701 : In response to a user clicking a shutter control, the camera application sends an image capture request to the camera service.

[0205] The camera application is in different shooting modes, such as high-brightness front-lit mode, high-brightness backlit mode, wide aperture mode, portrait mode, and night scene mode. The number of captured image frames required to change the exposure parameters for the multi-frame synthesis algorithm corresponding to the shooting mode may not be the same, and thus the number of image capture requests issued by the camera application may not be the same. This embodiment does not limit the number of image capture requests.

[0206] For example, the multi-frame synthesis algorithm requires 7 image preview frames and 2 captured image frames with changed exposure parameters, and the camera application sends two image capture requests in sequence.

[0207] As another example, the multi-frame synthesis algorithm requires 5 image preview frames and 4 captured image frames with changed exposure parameters, and the camera application sends four image capture requests in sequence.

[0208] The exposure parameters corresponding to different image capture requests are not necessarily the same, and this embodiment does not limit this.

[0209] S702: The camera service sends an image capture request to the camera HAL module.

[0210] The camera service receives the image capture request, performs the corresponding processing, such as creating a corresponding service instance, and sends the image capture request to the camera HAL module. The relevant processing of the camera service can be referred to in the existing technology and will not be repeated here.

[0211] S703: The camera HAL module receives the image capture request and stores the image capture request in the Bridge module.

[0212] Different from the image preview request, if the image capture request is received, the camera HAL module stores the image capture request in the Bridge module.

[0213] For example, if a camera application issues two image capture requests, such as image capture request 58 and image capture request 59, they are stored in the HAL's Bridge module, as shown in Figure 6a. Before and after this, image preview requests issued by the camera application are still stored in the HAL's request queue.

[0214] S704: The camera HAL module reads the image preview request from the request queue and sends the image preview request to the ISP module.

[0215] The camera HAL module receives and stores the request sent by the camera application, and the camera HAL module reads the image preview request from the request queue and sends it to the ISP module. This is an asynchronous process. This embodiment does not limit the execution order of S703 and S704.

[0216] S705: After receiving the image preview request, the ISP module reads whether there is an image capture request in the Bridge module.

[0217] S706 , if an image capture request exists in the Bridge module, the ISP module sequentially reads an image capture request from the Bridge module and sends the image capture request to the image sensor driver module.

[0218] S707: The ISP module discards the image preview request.

[0219] S708 , the image sensor driving module collects and captures image frames according to the capture parameters corresponding to the image capture request, and sends the captured image frames to the ISP module.

[0220] Continuing with the previous example, referring to Figures 6b and 6c, image preview request 56 and image preview request 57 are discarded, and image capture request 58 and image capture request 59 are prioritized in the form of queue interruption, shortening the output time of captured image frame 58 and captured image frame 59.

[0221] S709: The ISP module sends the captured image frame to the camera HAL module.

[0222] The ISP module can also perform corresponding image processing on the captured image frames, such as image denoising, image optimization, etc., which is not limited in this embodiment. Regarding the processing flow of the image frames by the ISP module, reference can be made to existing technologies and will not be repeated here.

[0223] S710: The camera HAL module stores the captured image frame in the ZSL queue.

[0224] S711: The camera HAL module obtains a corresponding number of preview image frames and captured image frames from the ZSL queue according to the algorithm corresponding to the photo-taking scene, performs a multi-frame fusion operation, and obtains the target image.

[0225] Continuing with the previous example, the camera HAL module sequentially stores the generated captured image frames 58 and 59 in the ZSL queue. Figure 8a illustrates an exemplary application scenario. As shown in Figure 8a, the image frames stored in the ZSL queue may include at least: preview image frame 49, preview image frame 50, preview image frame 51, preview image frame 52, preview image frame 53, preview image frame 54, preview image frame 55, captured image frame 58, and captured image frame 59.

[0226] In this example, the multi-frame synthesis algorithm requires 7 image preview frames and 2 captured image frames with changed exposure parameters. The camera HAL can use the corresponding algorithm to perform multi-frame fusion based on preview image frame 49, preview image frame 50, preview image frame 51, preview image frame 52, preview image frame 53, preview image frame 54, preview image frame 55, captured image frame 58 and captured image frame 59 to obtain the final target image to be output.

[0227] Image capture requests 58 and 59 are prioritized by being queue-jumped, while image preview requests 56 and 57 are discarded. This shortens the frame interval between preview image frame 55 and captured image frame 58, improving the continuity of these frames used for multi-frame fusion and enhancing the quality of the resulting composited image. In particular, in dynamic photography scenarios, a smaller frame interval between the captured and preview frames minimizes the image differences between them, resulting in a better-looking final composited image.

[0228] As another example, in a certain shooting mode, in response to the user clicking the shutter control, the camera application sequentially issues four image capture requests (such as image capture request 58, image capture request 59, image capture request 60, and image capture request 61) to change the exposure parameters. The camera HAL module then sequentially stores the generated captured image frames 58, 59, 60, and 61 in the ZSL queue. FIG8 b exemplifies an application scenario. As shown in FIG8 b , the image frames stored in the ZSL queue may include at least: preview image frame 49, preview image frame 50, preview image frame 51, preview image frame 52, preview image frame 53, preview image frame 54, preview image frame 55, captured image frame 58, captured image frame 59, captured image frame 60, and captured image frame 61.

[0229] In this example, the multi-frame synthesis algorithm requires 5 image preview frames and 4 captured image frames with changed exposure parameters. The camera HAL can use the corresponding algorithm to perform multi-frame fusion based on preview image frame 51, preview image frame 52, preview image frame 53, preview image frame 54, preview image frame 55, captured image frame 58, captured image frame 59, captured image frame 60 and captured image frame 61 to obtain the final target image to be output.

[0230] Image capture requests 58, 59, 60, and 61 are prioritized by being queue-jumped, while image preview requests 56 and 57 are discarded. This shortens the interval between preview image frame 55 and captured image frame 58, improving the continuity of these frames used for multi-frame fusion and enhancing the quality of the resulting composited image. In particular, in dynamic photography scenarios, the shorter the interval between the captured and preview frames, the smaller the difference between the captured and preview frames, resulting in a better final composited image.

[0231] S712: The camera HAL module sends the target image to the camera service.

[0232] S713: The camera HAL module generates a thumbnail of the target image and sends the thumbnail of the target image to the camera service.

[0233] This embodiment does not limit the execution order of S712 and S713.

[0234] S714: The camera service sends the target image to the gallery application.

[0235] S715 , the camera service sends a thumbnail of the target image to the camera application.

[0236] S716 , the gallery application stores the target image.

[0237] S717, the camera app refreshes the display of thumbnails.

[0238] Among them, in the above process, when the camera application displays the preview window, the preview image will also be refreshed in real time. The refresh process of the preview image and the refresh process of the thumbnail can refer to the existing technology and will not be repeated here.

[0239] For details not explained in detail in this process, please refer to the previous text and will not be repeated here.

[0240] The image sensor typically takes 33ms to expose a single frame of image data. By adding two preview frames to the queue for the captured image frame with the changed exposure parameters, the overall capture time can be reduced by 66ms. Furthermore, the intervals between frames used for multi-frame synthesis are smaller, resulting in a better-quality composite image.

[0241] It should be noted that, in the above embodiments, the camera application is used as an example for explanation, and the same is true for other third-party applications with camera functions, which will not be described in detail here.

[0242] This embodiment further provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the photographing method in the above-mentioned embodiment.

[0243] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the photographing method in the above-mentioned embodiment.

[0244] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the photographing method in the above-mentioned method embodiments.

[0245] Among them, the electronic devices (such as mobile phones, etc.), computer storage media, computer program products or chips provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0246] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0247] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0248] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A photographing method, characterized in that: Used in electronic equipment, including: Displaying an image preview interface of a target application; wherein the target application is an application having a camera function; and preview image frames displayed in the image preview interface are collected based on image preview requests continuously issued by the target application; In response to the received user operation, the target application generates an image capture request; wherein the exposure parameter of the image capture request is different from the exposure parameter of the image preview request; If there are both the unprocessed image preview request and the unprocessed image capture request, the image sensor is called to collect the captured image frame corresponding to the image capture request.

2. The method according to claim 1, characterized in that The image preview requests continuously sent by the target application are stored in the request queue, and the image capture requests sent by the target application are stored in the target module; Wherein, the request queue is a first-in-first-out queue.

3. The method according to claim 2, characterized in that The simultaneous existence of the unprocessed image preview request and the unprocessed image capture request includes: If the image preview request exists in the request queue and the image capture request exists in the target module, it is determined that both the unprocessed image preview request and the unprocessed image capture request exist.

4. The method according to claim 2 or 3, characterized in that The electronic device includes a camera hardware abstraction layer HAL module, an image signal processing ISP module and an image sensor driver module, and the method includes: The camera HAL module sequentially obtains one of the image preview requests from the request queue and sends the image preview request to the ISP module; The ISP module receives the image preview request; If the ISP module reads an image capture request from the target module, it sends the image capture request to the image sensor driver module, so that the image sensor driver module drives the image sensor to capture an image frame corresponding to the image capture request; If the ISP module does not read the image capture request in the target module, it sends the image preview request to the image sensor driving module, so that the image sensor driving module drives the image sensor to capture the preview image frame corresponding to the image preview request.

5. The method according to claim 4, characterized in that Also includes: If the ISP module reads an image capture request from the target module, it discards the image preview request.

6. The method according to claim 4, characterized in that Also includes: The camera HAL module receives an image preview request or an image capture request sent by the target application, stores the image preview request in the request queue, and stores the image capture request in the target module.

7. The method according to claim 4, characterized in that Also includes: The ISP module sends the preview image frame or the captured image frame fed back by the image sensor driving module to the camera HAL module; The camera HAL module sequentially stores the preview image frames or the captured image frames in a zero-second delay queue.

8. The method according to claim 7, characterized in that Also includes: In the scenario of multi-frame synthesis shooting, the camera HAL module reads a corresponding number of the preview image frames and the captured image frames in the zero-second delay queue, and uses an image processing algorithm matching the shooting mode to perform a multi-frame image synthesis operation to obtain a target image; wherein, the target image is used to store in the gallery application.

9. The method according to claim 1, characterized in that The target application is a camera application.

10. An electronic device, characterized in that: include: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the photographing method according to any one of claims 1 to 9.

11. A computer-readable storage medium comprising a computer program, characterized in that When the computer program is run on an electronic device, the electronic device is enabled to execute the photographing method according to any one of claims 1 to 9.

Images