Photography method and electronic device

Abstract

The present application provides a photography method and an electronic device. By means of the photography method, while the image output frame rate of a main road camera is maintained, the working intensity of a side road camera is reduced by reducing the image output frame rate of the side road camera in a preview stage, and while the quality of captured pictures is maintained, the power consumption of an electronic device during photography and the computing power burden on the electronic device are reduced.

Description

Shooting method and electronic equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on January 5, 2024, with application number 202410030549.9 and application name “Photographing method and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a shooting method and electronic equipment. Background Art

[0003] Nowadays, capturing and enhancing images has become an integral part of our daily lives and entertainment. Most electronic devices today use two cameras to capture photos, creating a variety of image effects. For example, one camera can capture the entire image, while the other captures depth of field, blurring the background outside the image's main subject to highlight it. However, using multiple cameras increases the power consumption of electronic devices and also maximizes their memory and computing power.

[0004] Therefore, how to reduce the power consumption of multiple camera devices during the shooting process is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a shooting method and electronic device, which reduces the working intensity of the auxiliary camera by reducing the frame rate of the auxiliary camera in the preview stage while maintaining the frame rate of the main camera. It reduces the power consumption of the electronic device during shooting and the burden on the computing power of the electronic device while maintaining the quality of the photos taken.

[0006] The above-mentioned and other objects are achieved by the features of the independent claims. Further implementations are given in the dependent claims, the description and the drawings.

[0007] In a first aspect, the present application provides a shooting method, comprising: in response to a first user operation, starting a portrait shooting mode, wherein the portrait shooting mode is a mode in which a first camera and a second camera are used to shoot together; controlling the first camera to generate a RAW image at a first frame rate, and controlling the second camera to generate a RAW image at a second frame rate, wherein the first frame rate is greater than the second frame rate.

[0008] In this application, in order for the electronic device to use a binocular algorithm (such as a binocular blur algorithm) to create a specific image effect for the image when the user subsequently takes the image, the electronic device needs to be able to simultaneously obtain the RAW images stored historically by both cameras after the user clicks to shoot. Therefore, in this application, the first camera and the second camera can simultaneously generate and output RAW images during the preview phase, and the output RAW images will be output to the cache.

[0009] In this method, the RAW image generated by the first camera can be processed and displayed for user preview, and the RAW image generated by the second camera can be used as auxiliary information to add specific image effects to the displayed image. In order to reduce the power consumption of the electronic device in the shooting scene while ensuring the image quality of the photo, the electronic device can control the frame rate of the RAW image generated by the first camera to be a larger frame rate, namely the first frame rate, and control the frame rate of the RAW image generated by the second camera to be a smaller frame rate, namely the second frame rate. In this way, when shooting images subsequently, the electronic device can not only select images with better image quality from a sufficient number of RAW images historically output by the first camera, but also select images from the RAW images historically output by the second camera to assist in implementing a binocular algorithm to create image effects for the image (for example, using a binocular blur algorithm to blur the image background), which can reduce the power consumption of the electronic device in the shooting scene while ensuring the image quality of the photo.

[0010] In an optional embodiment, the first user operation may be an operation in which the user opens a shooting software (such as a camera) and clicks on the portrait mode option to switch the electronic device to the portrait shooting mode. After entering the portrait shooting mode, the electronic device may continue to issue instructions for controlling the first camera and the second camera to generate RAW images. However, each request instruction generated by the electronic device will be issued to the first camera, but only one instruction will be issued to the second camera for each request instruction generated by the electronic device. Each time a request instruction generated by the electronic device is received, the camera will generate a frame of RAW image according to the instruction. Therefore, in the present application, the frame rate at which the first camera generates a RAW image is greater than the frame rate at which the second camera generates a RAW image.

[0011] In combination with the first aspect, in one possible implementation, the first frame rate is twice the second frame rate, and controlling the first camera to generate a RAW image at the first frame rate and controlling the second camera to generate a RAW image at the second frame rate includes: at a first moment, controlling the first camera to generate a first RAW image and controlling the second camera to generate a second RAW image; at a second moment, controlling the first camera to generate a third RAW image.

[0012] In this embodiment, the first and second moments correspond to two consecutive camera operating cycles, respectively, and at the second moment, the second camera does not output a RAW image. That is, for every two RAW frames generated and output by the image sensor in the first camera under the control of the electronic device, the image sensor in the second camera generates one RAW frame under the control of the electronic device. It is understood that while the second camera can also generate a larger number of frames (for example, the second camera generates one RAW frame for every three or four frames generated by the first camera) to further reduce the power consumption of the electronic device by the second camera, generating one RAW frame for every three or four frames generated by the first camera increases the probability of selecting a RAW image closer to the user's capture time when subsequently outputting the image. Consequently, the image content of the resulting photo based on the historically cached RAW images will more closely match the user's desired capture, minimizing the power consumption of the electronic device by the second camera while maximizing image quality.

[0013] In combination with the first aspect, in a possible implementation, at the first moment, before controlling the first camera to generate a first RAW image and controlling the second camera to generate a second RAW image, the method further includes: determining a first duration based on a fourth RAW image generated by the first camera and a fifth RAW image generated by the second camera, the fourth RAW image and the fifth RAW image having the same frame number, the first duration being the time interval between the moment when the first camera generates the fourth RAW image and the moment when the second camera generates the fifth RAW image; determining the first moment based on the first duration and the second frame rate.

[0014] However, in some embodiments, due to the performance limitations of the first and second cameras, although the electronic device can maintain the frame rate of the second camera generating the RAW image at 1 / 2 of the frame rate of the first camera generating the RAW image, the time when the second camera generates the RAW image may be significantly different from the time when the first camera generates the RAW image in the same working cycle. Therefore, in this embodiment, in order to ensure that the time when the second camera generates the RAW image is as close as possible to the time when the first camera generates the RAW image in the same working cycle, so that the electronic device can better identify the foreground and background of the image and more accurately blur the image, the electronic device can periodically select two frames of images with the same frame number from the primary and secondary RAW image cache queues, namely the fourth RAW image and the fifth RAW image, and determine the time interval between the times when the first and second cameras generate the two frames of images, namely the first duration, based on the timestamp information of the two frames of images. Since the frame rates of the RAW images output by the first and second cameras are fixed, the frame interval between the RAW images output by the first and second cameras is also relatively stable (without significant fluctuations). If the first duration is greater than a preset threshold (for example, greater than or equal to 1ms), the electronic device may adaptively adjust the frame interval between the second RAW image and the previous RAW image generated by the image sensor in the second camera before the second camera generates the latest RAW image (that is, the second RAW image). That is, based on the first duration and the specified frame interval, the electronic device may adaptively advance or delay the generation of the second RAW image that should have been generated at a certain time, so that the image sensor in the second camera and the image sensor in the first camera can respectively generate the latest RAW image frame at the same time.

[0015] In combination with the first aspect, in one possible embodiment, the method further includes: controlling the first camera to output a YUV image at the first frame rate, processing the first YUV image output by the first camera based on a monocular blur algorithm to obtain a first preview image; and displaying the first preview image in a preview area of ​​a display screen.

[0016] It is understandable that although the blur effect presented by the monocular blur algorithm is not as accurate as that presented by the binocular blur algorithm, the monocular blur algorithm has lower computing power requirements and power consumption for the device than the binocular blur algorithm, and what the user most needs to obtain is the captured photo rather than the preview image displayed during preview. Therefore, in this embodiment, the second camera may not output the YUV image during the preview stage. That is, the second camera can still output the RAW image to the cache of the electronic device, but the second camera will not output the YUV image converted from the RAW image. The images sent in the preview stream are all obtained by processing the YUV image output by the first camera based on the monocular blur effect. The power consumption of the electronic device can be reduced at the expense of the quality of the blur effect of the image in the preview stream.

[0017] In combination with the first aspect, in one possible implementation, the method is applied to a zero-second delay shooting mode, and the method further includes: saving a first photo in response to a user operation of a shooting control, where the first photo is obtained by processing at least one frame of a first RAW image generated and cached by the first camera and at least one frame of a second RAW image generated and cached by the second camera based on a binocular blur algorithm.

[0018] In this method, although the output frame rate of the RAW image generated by the second camera is reduced, the image front end in the second camera will still output and cache each RAW image frame generated and transmitted by the image sensor in the cache queue. Therefore, in this embodiment, after the user subsequently clicks the shooting control to take a photo, the electronic device can still obtain the RAW image cached by the primary camera (i.e., the first camera) and the RAW image cached by the auxiliary camera (i.e., the second camera) from the cache, and use the binocular blur algorithm to blur the two images, ensuring the blur effect of the first photo saved last.

[0019] In combination with the first aspect, in one possible embodiment, the first frame rate is n times the second frame rate, the RAW image generated by the first camera is output to the first cache queue, and the RAW image generated by the second camera is output to the second cache queue. When the first cache queue and the second cache queue are both full, the RAW images stored in the second cache queue are n times the number of RAW images in the first cache queue, where n is a positive number.

[0020] In order to be able to determine a pair of RAW images with the same generation time from the two RAW images output by the first camera and the second camera during the subsequent shooting and frame selection process, the electronic device will add frame numbers to the RAW images when outputting the RAW images to the cache. RAW images generated at the same time have the same frame numbers. When selecting RAW images, the electronic device can determine the required RAW images and the frame numbers corresponding to these RAW images from the RAW images output by the first camera, and then select RAW images with the same frame numbers from the RAW images output by the other camera based on the frame numbers corresponding to these RAW images. However, due to storage space limitations and the fact that the second camera outputs RAW images at intervals, when the user subsequently clicks the shooting control, when the electronic device selects the corresponding RAW images from the RAW images output by the first camera and the RAW images output by the second camera, after the electronic device determines multiple RAW frames from the RAW images output by the first camera, it may not be able to completely find the RAW images with the same frame numbers in the RAW images output by the other camera.

[0021] Therefore, in this embodiment, the electronic device can halve the length of the second cache queue to ensure that for any RAW image in the second cache queue, there is a RAW image with the same frame number in the first cache queue. The electronic device can then first determine the frame numbers of the required multiple RAW images from the second cache queue, and then find the multiple RAW images corresponding to the frame numbers of these multiple RAW images from the first cache queue, avoiding the situation where the corresponding RAW images cannot be found, causing the camera to re-frame.

[0022] In combination with the first aspect, in a possible embodiment, before saving the first photo, the method also includes: determining at least one frame of the second RAW image from the RAW image output by the second camera; determining at least one frame of the first RAW image from the RAW image output by the first camera based on the frame numbers of the at least two frames of the RAW image, and the number of the at least one frame of the first RAW image is the same as that of the at least one frame of the second RAW image; fusing the at least one frame of the first RAW image based on a multi-frame fusion algorithm to obtain a first image; processing the first key frame and the second key frame based on a binocular blur algorithm to obtain depth of field information of the first key frame, the first key frame being a frame of the at least one first RAW image with better image quality, and the second key frame being a RAW image with the same frame number as the first key frame in the at least one second RAW image; blurring the background of the first image based on the depth of field information to obtain the first photo.

[0023] In this embodiment, since any RAW image in the second cache queue has a RAW image with the same frame number as the RAW image in the first cache queue, the electronic device can first determine the at least one second RAW image from the RAW image output by the second camera, and it can also determine the same number of RAW images from the RAW image output from the first camera based on the frame numbers of the at least two RAW images, that is, the at least one first RAW image. In order to ensure the image quality of the final photo, the electronic device can determine a frame with better image quality in the at least one first RAW image as the first key frame based on the image quality assessment algorithm, and determine the RAW image in the at least one second RAW image with the same generation time as the first key frame as the second key frame. Based on the first key frame, the information of other images in the at least one first RAW image is superimposed on the first key frame to ensure the clarity of the subsequent photo. Afterwards, the electronic device can obtain the taken photos through the binocular blur algorithm, that is, calculate the depth of field information of each pixel point in the first key frame through the first key frame and the second key frame, and accurately determine the foreground and background of the first key frame based on the depth of field information, and then blur the background image. It can improve the clarity of the foreground image while accurately blurring the background image, avoiding the defects of missing background blur or mistaken foreground blur.

[0024] In combination with the first aspect, in one possible implementation, the first camera may be a wide-angle camera, and the second camera may be an ultra-wide-angle camera; or, the first camera may be a telephoto camera, and the second camera may be a wide-angle camera.

[0025] In the second aspect, the present application provides a shooting device, which includes a response unit, a first camera and a second camera. The response unit is used to start a portrait shooting mode in response to a first user operation, and the portrait shooting mode is a mode in which the first camera and the second camera are used to shoot together; the first camera is used to control the generation of RAW images at a first frame rate, and the second camera is used to generate RAW images at a second frame rate, and the first frame rate is greater than the second frame rate.

[0026] Specifically, the first camera and the second camera may be included in the same electronic device, and the first camera and the second camera may output RAW images and generate RAW images at the first frame rate and the second frame rate, respectively, under the control of the electronic device. Alternatively, the first camera and the second camera may include program code or logic circuitry, and the first camera and the second camera may control the frame rate of the RAW images they generate based on the program code or logic circuitry contained therein.

[0027] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method in the first aspect or any possible implementation of the first aspect.

[0028] In a fourth aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the processors are used to call computer instructions to enable the electronic device to execute the method in the first aspect or any possible implementation of the first aspect.

[0029] In a fifth aspect, a computer program product comprising instructions, when the computer program product is run on an electronic device, enables the electronic device to execute the method in the first aspect or any possible implementation of the first aspect.

[0030] In a sixth aspect, a computer-readable storage medium is provided, comprising instructions. When the instructions are executed on an electronic device, the electronic device executes the method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] FIG1 is a schematic diagram of the architecture of a camera system provided in an embodiment of the present application;

[0032] FIG2 is a schematic diagram of the architecture of a camera system provided in an embodiment of the present application;

[0033] FIG3 is a flow chart of a photographing method provided in an embodiment of the present application;

[0034] Figures 4 to 6 are some user interface diagrams provided in the embodiments of the present application;

[0035] FIG7 is a schematic diagram of a method for outputting an image by an electronic device according to an embodiment of the present application;

[0036] FIG8 is a schematic diagram of a RAW image cache queue provided in an embodiment of the present application;

[0037] FIG9 is a schematic diagram of a scenario in which a primary and secondary camera generates and outputs a RAW image according to an embodiment of the present application;

[0038] FIG10 is an Android system architecture diagram provided in an embodiment of the present application;

[0039] FIG11 is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The terms used in the following examples of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and encompasses any or all possible combinations of one or more of the listed items.

[0041] To facilitate understanding, the following first introduces the relevant terms involved in the embodiments of this application.

[0042] (1) Multi-device

[0043] A multi-camera device refers to an electronic device with two or more cameras. For example, the electronic device provided in this application may include multiple cameras, which may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, etc., and may also include other types of cameras. These cameras can be placed on the front or back of the electronic device, that is, as the front camera or rear camera of the electronic device.

[0044] Compared with monocular devices (electronic devices with only one camera), multi-eye devices have a wider shooting range and more viewing angle options, can capture more comprehensive scenes and multi-dimensional images, and have a field of view greater than traditional monocular devices. Multi-eye devices can synthesize the fields of view of multiple lenses to obtain a wider range of images. Specifically, multi-eye devices can use multiple camera modules to form images separately, and through specific algorithm processing, the images captured by multiple modules are fused into one image to achieve specific imaging needs. Secondly, multi-eye camera devices can achieve depth perception, that is, combined with depth measurement technology, allowing users to better perceive the depth of the scene while recording the image. For example, when shooting portraits, multi-eye devices can blur the background based on distance data or make the portrait more realistic while blurring.

[0045] In this application, the image data output by each camera in a multi-eye device can be referred to as a data stream. When a user uses a multi-eye device to frame a target, the data stream output by one camera on the multi-eye device can be used for display (i.e., displayed on the screen of the electronic device for user preview), and the data streams output by other cameras can be used for auxiliary information detection, alignment, ranging and other functions to correct or beautify the image in the above-mentioned data stream for display, thereby improving the quality of the displayed image. Among them, the camera whose output data stream is used for display can be referred to as the "main camera", and the camera whose output data stream is used for auxiliary information detection, alignment, and ranging can be referred to as the "auxiliary camera".

[0046] However, the simultaneous output of data streams by multiple cameras will undoubtedly increase the power consumption of electronic devices and maximize the use of computing resources and memory resources of electronic devices.

[0047] (2) Background blur

[0048] Background blur, also known as depth of field effect, is a camera technology that makes the depth of field shallower, so that the lens focuses on the main subject of the picture and the background appears blurred. It is most commonly used in the portrait mode provided by camera software. When shooting objects in portrait mode, electronic devices can automatically blur the background to highlight the subject.

[0049] Currently, the background blurring technologies commonly used in electronic devices include monocular blurring and binocular blurring.

[0050] Monocular bokeh refers to a technology that achieves a blur effect using only a single camera. With monocular bokeh, electronic devices can control the degree and range of blur by adjusting the camera's focal length and aperture. As you can imagine, when the camera's focal length is farther away, the aperture is smaller, the depth of field is greater, and the entire image appears clearer. Conversely, when the camera is closer, the aperture is larger and the depth of field is smaller, so only objects near the focal point appear sharp, while other areas appear blurred. In this way, electronic devices can achieve a blur effect by adjusting the camera's focal length and aperture to emphasize the main subject in the image while blurring the background.

[0051] Binocular blurring requires technology that uses two cameras to achieve a blurring effect, so it needs to be applied to multi-eye devices. In addition, due to the limitations of the triangulation principle, the two camera modules need to be calibrated so that the imaging planes of the two are on the same plane and the pixels are aligned. In other words, the two cameras used for shooting need to be on the same side of the electronic device, that is, both cameras need to be the front cameras of the electronic device or the rear cameras of the electronic device. In a dual-camera system, one camera is responsible for shooting the subject, and its output image data stream can be used for display; the other camera is responsible for measuring the depth of field information of each point, that is, the electronic device can determine the distance from each object in the scene to the electronic device (camera) through the image captured by the camera, and separate the foreground and background according to this information, and then use the blurring algorithm to blur the background. Compared with monocular blurring, binocular blurring can be more precise and delicate in blurring the subject, thereby simulating the visual effect of the human eye to a greater extent, making the blurring effect more natural.

[0052] (3) RAW image and YUV image

[0053] A RAW image is the raw data generated by an image sensor converting the captured light source signal into a digital signal. It is the data format in which the image sensor outputs the image. A sensor, also known as a photosensitive element, is a device that converts optical images into electronic signals, such as a charge coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) phototransistor. In the field of imaging technology, RAW format refers to an unprocessed, uncompressed format, representing the original image encoding data (digital negative). Common RAW format suffixes include .ARW, .SRF, .SR2, .crw, .cr2, and .cr3.

[0054] YUV is a digital image format that separates brightness (Y) and color (U, V) information. It is commonly used in video encoding and digital image processing. YUV is actually a color encoding method that separates brightness information (Y) from color information (UV). While it can display a complete image without UV information, the image will be black and white.

[0055] It's important to understand that the RAW images output by the sensor are generally not directly displayed on electronic device displays. This is because the human eye cannot directly perceive scene information from RAW images. This is because CMOS sensors are generally single-channel image sensors, with each pixel recording only one channel of signal. Therefore, the RAW images output by the sensor must be processed by electronic devices through a series of operations, including white balance correction, color space conversion, and tone mapping, before the YUV images can be displayed on electronic device displays.

[0056] (4) Zero shutter lag (ZSL)

[0057] In daily life, taking photos with a mobile phone often experiences some delay. After the user taps the capture control on their phone, there's usually a short wait before the device displays the captured image. This is because in non-ZSL mode, the device begins previewing and displays a preview frame. When the user presses the shutter button, the preview stream stops, and the device undergoes a series of actions, including focusing, preparing to capture, exposing, and processing the capture stream data. The resulting image is a RAW image. The device then encodes the RAW image to a JPEG image and saves it to a designated storage area. At this point, the capture is complete, and preview resumes. In other words, in non-ZSL mode, previewing stops while a photo is being taken, and the RAW images used in the preview stream are not stored in the device. Only after the user presses the shutter button does the device store and process several frames of the RAW image to produce a viewable image (e.g., a JPEG image). For example, when a user is photographing a jumping target, the user wants to capture the moment the target jumps and press the shutter button at the moment the target jumps. However, in non-ZSL mode, the electronic device can only use the RAW image captured by the camera after the user presses the shutter button to process the final photo. After the user presses the shutter button, the target may have already jumped and returned to the ground. Therefore, the RAW image captured by the camera after the user presses the shutter button is not the image captured at the moment the target jumps. The final photo is likely to show not the target jumping, but the target landing. In other words, in non-ZSL shooting mode, there is a certain delay between the picture captured by the user and the picture the user actually wants to shoot.

[0058] ZSL is a photo mode developed to eliminate this delay and provide a "shoot and see" experience. In ZSL mode, when the electronic device begins previewing, the raw image generated by the sensor is stored in a cache. When the user presses the shutter, the device's system calculates the actual capture time, locates the RAW image of the corresponding frame stored in the cache, and then encodes the RAW image to generate a JPEG image, which is saved to a designated storage area. In other words, in ZSL mode, previewing continues when the user presses the shutter. The RAW image corresponding to the most recently used image in the preview stream is stored in the cache. After the user presses the shutter, the system calculates the delay (the time difference between the user's desired capture time and the time the user actually presses the shutter, i.e., the user's reaction time), identifies a specific image frame as the real-time capture frame, and processes it to generate a viewable image (e.g., a JPEG image).

[0059] (5) Camera structure

[0060] In the present application, each camera of the electronic device may include an image sensor (sensor) and an image processor (image signal processing, ISP). Among them, the sensor is the photosensitive element in the above description, which can convert photons into electronic signals, and convert them into digital signals through an amplification circuit and an analog-to-electric conversion circuit, that is, a RAW image. The ISP may include an image front end (IFE). During the operation of the camera, the RAW image output by the sensor will first be transmitted to the IFE, and the IFE can perform color correction, downsampling, demosaicing and other processing on the RAW image to obtain a YUV image; the IFE will further output the YUV image to the image processing engine (IPE), and the IPE will perform hardware noise reduction, cropping, noise reduction, color processing, detail enhancement and other image processing on the YUV image to obtain the final YUV image for display (this image is an image for user preview, not a photo obtained after the user clicks the shooting control). If the current shooting mode of the electronic device is ZSL mode, the IFE will further store the received RAW image in the cache. When the user clicks the shooting control, the electronic device can calculate the actual shooting time, find the RAW image of the corresponding frame stored in the cache, and then use the RAW image to encode it to obtain a JPEG image and save it to the set storage area. The JPEG image is the photo obtained by taking the photo.

[0061] It should be understood that in addition to the above-mentioned components, each camera in this application may also include other components, such as lenses, focus motors, filters, etc., and this application does not limit this.

[0062] (6) Frame rate

[0063] Frame rate is the frequency (rate) at which a bitmap image, in units of frames, appears continuously on a display. In an embodiment of the present application, the frame rate can represent the number of images generated per second by a camera (or camera element). For example, it can be used to represent the number of RAW images generated per second by a sensor in a camera. Specifically, assuming that the first sensor in the first camera in a multi-camera device generates a RAW image every 0.02 seconds, and the second sensor in the second camera generates a RAW image every 0.04 seconds, it can be considered that the frame rate at which the first camera (first sensor) generates a RAW image is twice that of the second camera (second sensor) generating a RAW image.

[0064] (7) Application layer, application framework layer, hardware abstraction layer

[0065] The application layer (Application), application framework layer (Android Framework), and hardware abstraction layer (Hardware Abstraction Layer) are the four layers in the Android system architecture. Among them:

[0066] The application layer includes built-in applications and non-system-level applications, which are usually developed based on the Java language and are mainly responsible for direct interaction with users. For example, when a user operates an electronic device running on the Android system and opens a photo-taking application (such as a camera), the application layer can respond to the user operation and send the request to the application framework layer through the Camera Api v2 standard interface, and wait for the application framework layer to return the processing results, which include image data and overall camera system status parameters, and then feed the results back to the user in a certain way, such as using the SurfaceFlinger module to arrange the position and content of each layer in the image returned by the application framework layer, so that the image is finally displayed on the screen of the electronic device.

[0067] The application framework layer is the foundation of Android application development. Written in Java, it provides developers with the APIs they need to build applications. Many core applications also rely on this layer to implement their core functionality. This layer simplifies component reuse, allowing developers to quickly develop applications using the provided components directly or extend them through inheritance. Specifically, the application framework layer includes multiple components, such as the activity manager, window manager, content providers, view system, package manager, telephony manager, resource manager, location manager, and notification manager.

[0068] The hardware abstraction layer encapsulates the underlying hardware driver and provides a general interface for calling the driver to the application framework layer. As long as the manufacturer implements the corresponding interface according to the specification and stores it in a specific directory in the form of a shared library, the upper layer only needs to load this shared library and find the pointer to the device corresponding to the corresponding module. After obtaining the pointer to the entire device, it can operate the underlying hardware. For example, the aforementioned application framework layer can send the frame request instruction received from the application layer to the hardware abstraction layer, so that the sensor in the hardware abstraction layer generates a frame of RAW image according to the instruction, and passes the RAW image to the IFE. After the IFE converts it into a YUV image, the YUV image is further passed to the IPE for processing. The final image can be returned to the application layer through the application framework layer for display.

[0069] To achieve superior image quality and create a variety of visual effects, most electronic devices today are typically equipped with multiple cameras. For example, a mobile phone's rear camera may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, and so on. In many shooting scenarios, multi-camera devices use two cameras simultaneously. For example, in portrait mode, a multi-camera device can capture the entire scene with one camera, and the data stream output by this camera is sent to the display. Simultaneously, the multi-camera device can also obtain depth of field information from a second camera, blurring the background outside the image's main subject, thereby highlighting the main subject.

[0070] FIG1 shows the working mode of the camera system 10 in the multi-camera electronic device in the ZSL mode and the portrait mode.

[0071] As shown in FIG1 , the camera system 10 may include a camera 101, a camera 102, and an image processing engine 103. The camera 101 may include an image sensor 1011 and an image front end 1012, and the camera 102 may include an image sensor 1021 and an image front end 1022. In the embodiment of the present application, the camera 101 may be referred to as a primary camera, and the camera 102 may be referred to as an auxiliary camera. In some embodiments, the image processing engine 103 may work in conjunction with the camera 101 and the camera 102 respectively under the control of software code. The processing logic of the image processing engine 103 when working in conjunction with the camera 101 and the processing logic when working in conjunction with the camera 102 may be different, and the two may be performed simultaneously without affecting each other.

[0072] After the user opens a shooting application (such as a camera), the camera 101 and the camera 102 can output images at the same time.

[0073] Specifically, the image sensor 1011 of the camera 101 can collect light signals at a certain acquisition rate, convert the light signals into RAW images, and send the generated RAW images to the image front end 1012. Since the electronic device needs to use the RAW images stored in the cache to generate photos in the ZSL shooting mode, after the image front end 1012 receives the RAW images transmitted by the image sensor 1011, on the one hand, the image front end 1012 will perform color correction, demosaicing and other processing on the received RAW images, convert the RAW images into YUV images, and transmit the YUV images to the image processing engine 103 at a frame rate of fps1 (that is, the image front end 1012 sends fps1 frames of RAW images to the image processing engine 103 per second, the same below); on the other hand, the image front end 1012 will also output the RAW images to the cache of the electronic device at a frame rate of fps1, that is, the main RAW image cache queue in Figure 1. After receiving the YUV image sent by the image front end 1012, the image processing engine 103 will further perform noise reduction, cropping and other processing on the YUV image to obtain the final main YUV image for display.

[0074] At the same time, the image sensor 1021 of the camera 102 can also collect light signals at a certain collection rate, convert the collected light signals into RAW images, and send the generated RAW images to the image front end 1022. Similarly, after the image front end 1022 receives the RAW image transmitted by the image sensor 1021, the image front end 1022 will process and convert the received RAW image into a YUV image, and also send the YUV image to the image processing engine 103 at a frame rate of fps1; on the other hand, the image front end 1022 will output the RAW image to the cache of the electronic device at a frame rate of fps1, that is, the auxiliary RAW image cache queue in Figure 1. After receiving the YUV image sent by the image front end 1022, the image processing engine 103 will further perform noise reduction, cropping and other processing on the YUV image to obtain an auxiliary YUV image used to assist in the display of the aforementioned main YUV image.

[0075] It should be noted here that the number of RAW images generated per second by the image sensor 1011 is the same as the number generated per second by the image sensor 1021, and the number of RAW images sent per second by the image sensor 1011 to the image front end 1012 is also the same as the number of RAW images sent per second by the image sensor 1021 to the image front end 1022; in addition, the image front end 1012 and the image front end 1022 will send YUV images to the image processing engine 103 and the image processing engine 103 respectively at the same frame rate fps1, and output the RAW images to the cache respectively at the same frame rate fps1.

[0076] During the real-time preview phase, the electronic device can further process the primary YUV image output by the image processing engine 103 using a binocular defocusing algorithm to obtain an image with a defocusing effect that is ultimately displayed on the screen for user preview. Specifically, the electronic device can use the primary YUV image output by the image processing engine 103 as a reference, and in combination with the YUV image output by the image processing engine 103, calculate the depth of field information of each pixel in the primary YUV image, thereby determining the foreground and background in the primary YUV image, and defocusing the background of the primary YUV image. The resulting YUV image after defocusing can then be transmitted to the screen of the electronic device for user preview.

[0077] During the shooting stage, that is, after the user clicks the shooting control (shutter), the electronic device can select several frames of historically cached main RAW images from the main RAW image cache queue according to the shooting time (that is, the time when the user clicks the shooting control), and based on the selected main RAW images, select RAW images with the same number of frames and the same main RAW image generation time from the auxiliary RAW image cache queue, and then blur them and fuse multiple frames to obtain the final photo stored in the electronic device.

[0078] Optionally, the electronic device can determine three RAW images P1, P2 and P3 from the above-mentioned main RAW image cache queue, and determine the corresponding three RAW images P1', P2' and P3' from the above-mentioned auxiliary RAW image cache queue based on P1, P2 and P3, wherein the numbers of the caches used to store P1, P2, P3 are respectively the same as the numbers of the caches used to store P1', P2', P3' (that is, the image sensor 1011 and the image sensor 1021 generate P1 and P1' respectively for the exposure processing of the shooting scene at the same time, and generate P2 and P2' respectively for the exposure processing of the shooting scene at another time, and so on). The electronic device can calculate the depth of field information of each pixel point in P1 based on P1, through P1 and P1', and determine the foreground and background of P1 based on the depth of field information, and then blur the background in P1 to obtain a frame of image with a blurred effect; similarly, the electronic device can obtain another two images with a blurred effect based on P2 and P2', P3 and P3'. Afterwards, the electronic device can use one of the images with the blur effect as a key frame, and superimpose the information in the other two images with the blur effect onto the key frame through a multi-frame fusion algorithm. The final image can be stored in the electronic device as a photo.

[0079] Optionally, in some embodiments, after the user clicks the shooting control, the image sensor 1011 and the image sensor 1021 can continue to output RAW images and continue to transmit them to the image front end 1012 and the image front end 1022 respectively. The image front end 1012 and the image front end 1022 can also continue to output the received RAW images respectively and store them in the above-mentioned main RAW image cache queue and the above-mentioned auxiliary RAW image cache queue. Afterwards, the electronic device can determine six RAW images P1, P2, P3, P4, P5, and P6 from the above-mentioned main RAW image cache queue (wherein P1, P2, and P3 are RAW images cached before the user clicks the shooting control, and P4, P5, and P6 are RAW images cached after the user clicks the shooting control), and based on P1, P2, P3, P4, P5, and P6, determine the corresponding six RAW images P1', P2', P3', P4', P5', and P6' from the above-mentioned auxiliary RAW image cache queue (wherein P1', P2', and P3' are RAW images cached before the user clicks the shooting control, and P4', P5', and P6' are RAW images cached after the user clicks the shooting control); afterward, The electronic device can determine the key frame from the six RAW images of P1, P2, P3, P4, P5, and P6 through image quality evaluation (here it is assumed that P1 has the best image quality and is determined as the key frame), and use the remaining frames (i.e., P2-P6) as reference frames. First, the information in all reference frames is superimposed on the key frame through a multi-frame fusion algorithm to obtain a frame of image P7 fused from multiple frames, and then the depth of field information of each pixel point in P1 is calculated through P1 and P1' (i.e., the reference frame and the auxiliary RAW image corresponding to the reference frame), and the foreground and background of P7 are determined based on the depth of field information, and then the background in P7 is blurred to obtain an image P8 with a blurred effect. The image P8 is stored in the electronic device as the photo taken. Optionally, after the user clicks the shooting control, the electronic device can use other methods to process the historically cached RAW images to obtain the above-mentioned photos. The specific method can be determined by the algorithm logic of the multi-frame fusion algorithm and the binocular blur algorithm adopted by the electronic device, and this application does not limit this. However, in order to ensure the background blur effect of the photo and there is no excessive delay between the photo image and the time when the user takes the photo, the historical cached RAW image obtained by the electronic device (that is, the RAW image stored before the user clicks the shooting control) must contain both the historical cached RAW image of camera 101 and the historical cached RAW image of camera 102.

[0080] Combined with the above description, it can be seen that in ZSL shooting mode, in order to achieve a more accurate background blur effect for the image in portrait mode, the two cameras contained in the multi-eye electronic device will use binocular blur technology to blur the image during preview and shooting. In addition, when previewing, both cameras need to output YUV images at the same frame rate for the user to preview, and store the RAW images in the cache queue at the same frame rate, so that the electronic device can select frames and output images with zero delay after the user clicks the shooting control. However, for multi-eye electronic devices, the continuous collaborative work of multiple cameras will increase the power consumption of the electronic device, especially when using multiple cameras to shoot portraits. The links involved in the portrait algorithm are too long, which puts a greater burden on the memory resources and computing power of the electronic device.

[0081] In response to the above problems, the present application provides a shooting method, which reduces the working intensity of the auxiliary camera by reducing the frame rate of the auxiliary camera during the preview stage while ensuring the frame rate of the main camera, thereby reducing the power consumption of the electronic device during shooting and the burden on the computing power and memory of the electronic device.

[0082] It should be noted in advance that although the blurring effect presented by the monocular blurring algorithm is not as precise as that presented by the binocular blurring algorithm, the monocular blurring algorithm requires less computing power and power consumption than the binocular blurring algorithm. In addition, the user ultimately needs to obtain the captured photo rather than the preview image displayed during the preview. Therefore, in this method, the auxiliary camera reduces the output frame rate during the preview phase in two aspects: 1. During the preview phase, the auxiliary camera reduces the output frame rate of the RAW image, that is, reduces the rate at which the image sensor in the auxiliary camera generates the RAW image; accordingly, the frame rate at which the image sensor transmits the RAW image to the image front end and the frame rate at which the image front end outputs the RAW image and caches it in the cache queue will also be reduced. 2. During the preview phase, the auxiliary camera does not output the YUV image. That is, the image sensor of the auxiliary camera can still output the RAW image to the image front end, but the image front end will not output the converted YUV image to the image processing engine, and the image processing engine (auxiliary camera) will not output the YUV image. The images sent in the preview stream are all obtained by processing the YUV image output by the main camera based on the monocular blur algorithm, that is, sacrificing the quality of the image in the preview stream to reduce the power consumption of the electronic equipment.

[0083] It should be understood that in the embodiment of the present application, the frame rate at which the image sensor in the auxiliary camera generates a RAW image is only lower than the frame rate at which the image sensor in the main camera generates a RAW image. However, after the image sensor in the auxiliary camera generates a RAW image, it will still transmit the generated RAW image to the image front end of the auxiliary camera, and the image front end of the auxiliary camera will also output all the RAW images it receives to the cache queue. In this way, in ZSL mode, even if the electronic device processes the YUV image output by the main camera based on the monocular blur algorithm during the preview phase to obtain the preview image, during the photo-taking phase, the electronic device can still obtain the historical cached image of the auxiliary camera from the cache queue, and the electronic device can then use the binocular blur algorithm to obtain photos with better blurring effects.

[0084] FIG2 shows the specific working mode of the camera system of the electronic device provided by the present application in the ZSL mode. As shown in FIG2 , the camera system 20 may include a camera 201, a camera 202, and an image processing engine 203. The camera 201 may include an image sensor 2011 and an image front end 2012, and the camera 202 may include an image sensor 2021 and an image front end 2022. In some embodiments, the image processing engine 203 may work in conjunction with the camera 201 and the camera 202 respectively under the control of the software code. The processing logic of the image processing engine 203 when working in conjunction with the camera 201 and the processing logic of the image processing engine 203 when working in conjunction with the camera 202 may be different, and the two may be performed simultaneously without affecting each other.

[0085] Among them, camera 201 can be called the main camera, and camera 202 can be called the auxiliary camera. Camera 201 and camera 202 are both front cameras, or camera 201 and camera 202 are both rear cameras. This application does not limit the types of camera 202 and camera 201. Specifically, camera 201 can be a wide-angle camera, and camera 202 can be an ultra-wide-angle camera; or, camera 201 can be a telephoto camera, and camera 202 can be a wide-angle camera. After the user opens a shooting application (such as a camera), camera 201 and camera 202 can work in different ways.

[0086] Specifically, the image sensor 2011 of the camera 201 collects light signals at a certain acquisition rate and converts the light signals into RAW images at a frame rate of fps2. That is, the image sensor 2011 generates fps2 frames of image per second and continuously transmits these RAW images to the image front end 2012. After receiving the RAW images transmitted by the image sensor 2011, the image front end 2012 performs color correction, demosaicing and other processing on the received RAW images, converts the RAW images into YUV images, and sends the YUV images to the image processing engine 203 at the same frame rate of fps2. The image processing engine 203 performs noise reduction, cropping and other processing on the YUV images to obtain the final main YUV images for display. In addition, the image front end 2012 also outputs the RAW images at a frame rate of fps2 to the cache of the electronic device, namely the main RAW image cache queue in Figure 2.

[0087] While camera 201 is operating, camera 202 can also operate. Specifically, image sensor 2021 in camera 202 can also collect light signals at a certain acquisition rate and convert the collected light signals into RAW images at a frame rate of fps3. That is, image sensor 2021 will generate fps3 frames of image per second and continuously transmit these RAW images to image front end 2022. After image front end 2022 receives the RAW image transmitted by image sensor 2021, it can also process the RAW image and convert it into a YUV image. However, image front end 2022 may not send the converted YUV image to image processing engine 203 (at this time, the YUV image transmission channel between image front end 2022 and image processing engine 203 is closed); however, image front end 2022 will output the RAW image at a frame rate of fps3 to the cache of the electronic device, namely, the auxiliary RAW image cache queue in Figure 2.

[0088] Specifically, the electronic device can control image sensor 2011 and image sensor 2021 to generate RAW images at different frame intervals, wherein the frame interval at which image sensor 2011 generates RAW images is smaller than the frame interval at which image sensor 2021 generates RAW images. For example, under the control of the electronic device, image sensor 2011 can generate a RAW image frame every 1 / 30 second, while image sensor 2012 can generate a RAW image frame every 1 / 15 second. Both image sensor 2011 and image sensor 2021 can transmit each generated RAW image frame to image front end 2012 and image front end 2022, respectively. It can be understood that since the frame rates of images generated by the image sensors contained in the two cameras are different, the number of RAW images sent by the image sensor 2011 to the image front end 2012 per second and the number of RAW images sent by the image sensor 2021 to the image front end 2022 per second are also different; similarly, the number of RAW images sent by the image front end 2012 and the image front end 2022 to the image processing engine 203 per second are also different, which are both determined by the frame rates of images generated by the image sensors contained in the two cameras.

[0089] Optionally, the value of the fps3 can be 1 / 2 of the value of the fps2. Alternatively, the value of the fps3 can also be 1 / 3 of the value of the fps2, or other values ​​smaller than the value of the fps2, which is not limited in this application. Therefore, it can be understood that the number N of RAW images in the main RAW image cache queue in Figure 2 will be greater than the number n of RAW images in the auxiliary RAW image cache queue in Figure 2. Taking the value of fps3 as 1 / 2 of the value of fps2 as an example, within the same time length, the above N is equal to 2n.

[0090] During the real-time preview phase, the electronic device can further process the primary YUV image output by the image processing engine 203 using a monocular defocusing algorithm to obtain a final image with a defocusing effect for display on the screen for user preview. Specifically, the electronic device can use an AI algorithm to identify the subject in the primary YUV image, defocus the background, and transmit the resulting defocused YUV image to the screen of the electronic device for user preview.

[0091] During the shooting phase, the electronic device can select several frames of historically cached RAW images from the primary RAW image cache queue and the secondary RAW image cache queue based on the shooting time (i.e., the time when the user clicks the shooting control), and then perform blur processing and then fuse the multiple frames of blurred images to obtain the final photo stored in the electronic device. For details, please refer to the above description of FIG. 1 and the description of the subsequent embodiments, which will not be repeated here.

[0092] Of course, in some embodiments, if a preview image with a better blur effect is also required to be displayed to the user during the real-time preview phase, the electronic device can also control the image front end 2022 in the auxiliary camera (i.e., camera 202) to transmit a YUV image to the image processing engine 203 during the preview phase (the YUV image transmission channel between the image front end 2022 and the image processing engine 203 is in an open state at this time). By simply adjusting the frame rate of the RAW image generated by the auxiliary camera, the frame rate of the RAW image generated by the auxiliary camera is made lower than the frame rate of the RAW image generated by the main camera. In this way, the electronic device can provide the user with a preview image with a better blur effect during the preview phase, and can also reduce the working intensity of the auxiliary camera to a certain extent, thereby reducing the power consumption of the electronic device.

[0093] In combination with the above description of the camera system 20 , the shooting method provided by this application is introduced below, please refer to FIG3 .

[0094] Figure 3 is a flow chart of a shooting method provided by an embodiment of the present application. Figure 3 illustrates the shooting method provided by the present application by taking the example of a method in which the preview stream adopts a monocular blur algorithm, the photo adopts a binocular blur algorithm, and the frame rate of the RAW image generated by the auxiliary camera is half the frame rate of the RAW image generated by the main camera. As shown in Figure 3, in the shooting method provided by the embodiment of the present application, the first acquisition module and the first IFE module can be included in the first camera, and the first camera can be the main camera in this method; the second acquisition module and the second IFE module can be included in the second camera, and the second camera can be the auxiliary camera in this method. Among them, the first camera can be the aforementioned camera 201, and the second camera can be the aforementioned camera 202. The present application does not limit the specific types of the first camera and the second camera. Specifically, the first camera can be a wide-angle camera, and the second camera can be an ultra-wide-angle camera; or, the first camera can be a telephoto camera, and the second camera can be a wide-angle camera.

[0095] When implementing this method, the camera application of the application layer can load the portrait mode in response to the user's operation of opening the camera application. After the portrait mode is loaded, the user can start the portrait mode by touching the portrait mode icon. Then, the hardware abstraction layer can identify the shooting scene and report it to the shooting control module of the application layer. The shooting control module can adjust the shooting parameters and shooting mode in the portrait mode and send it back to the first acquisition module and the second acquisition module of the hardware abstraction layer. Finally, the first acquisition module and the second acquisition module can capture images according to the adjusted shooting parameters and shooting mode. The first IFE module, the second IFE module and the IPE module can determine the image processing algorithm to be used according to the identified shooting scene, and use the image processing algorithm to process the captured image. The image data stream obtained after processing can be encoded to obtain an image file. The preview display module can also obtain the processed image data stream for preview display.

[0096] In this method, the IPE module can work in conjunction with the first camera and the second camera respectively under the control of the software code. The processing logic when the IPE module works in conjunction with the first camera and the processing logic when the IPE module works in conjunction with the second camera can be different, and the two can be carried out simultaneously without affecting each other.

[0097] It is understood that in Figure 3, data (including instructions, information, and images) between the application layer and the hardware abstraction layer can be transmitted via the application framework layer (Android Framework). For ease of explanation, Figure 3 omits the information transmission process of the application framework layer. The shooting method may include but is not limited to the following steps:

[0098] S101: The user starts a camera application.

[0099] In the embodiment of the present application, the user can start the camera application by operating the application icon of the camera application, such as touching it.

[0100] S102: The mode loading module loads the mode.

[0101] When the camera application is started, the mode loading module can query the hardware abstraction layer for the mode. In this embodiment of the present application, the hardware abstraction layer can provide a portrait mode for the camera application. That is, in portrait mode, the first and second acquisition modules, the first IFE module and the IPE module, and the second and second IFE modules in the hardware abstraction layer can be activated to perform their respective functions.

[0102] In an embodiment of the present application, the hardware abstraction layer can also provide other modes for the camera application, such as normal mode, night mode, and video mode, etc., which is not limited in this embodiment of the present application.

[0103] Specifically, the mode loading module can query the hardware abstraction layer for a mode. In response to the query, the hardware abstraction layer can provide the module with the modes provided by the hardware abstraction layer for the camera application. For example, the modes provided include portrait mode, normal mode, night mode, and video mode.

[0104] Among them, the loaded modes include portrait mode, and the mode loading module also initializes the modules corresponding to each mode in the application layer and the hardware abstraction layer during the loading process. After initialization, the electronic device can display the icon corresponding to each mode, and please refer to the subsequent instructions for details. After initialization, in response to the user's touch operation on the icon corresponding to the portrait mode, the shooting control module can notify the first acquisition module, the second acquisition module, the first IFE module, the second IFE module and the IPE module in the hardware abstraction layer to start to perform their respective functions. After initialization, other modes are similar to the portrait mode, and can start the corresponding modules in the hardware abstraction layer in response to the user's touch operation on the icon corresponding to the mode.

[0105] The following describes the user interface involved in the process of loading the portrait mode with reference to FIG4 and FIG5.

[0106] Please refer to Figure 4, which is a schematic diagram of a human-computer interaction interface provided in an embodiment of the present application. As shown in (A) in Figure 4, the electronic device can display a user interface 11. The user interface 11 includes a calendar widget 111, a weather widget 112, an application icon 113, a status bar 114, and a navigation bar 115. Among them, the application icon 113 can include, a gallery icon, and a camera icon 1131, etc., and can also include icons of other applications, which is not limited in this embodiment of the present application. Any application icon can be used to respond to a user's operation, such as a touch operation, so that the electronic device starts the application corresponding to the icon.

[0107] The user can start the camera application by touching the camera icon 1131. As shown in Figure 4 (A), in response to the user touching the camera icon, the mode loading module executes step S102. After the mode loading module completes loading the mode, the electronic device can display the icon corresponding to each mode.

[0108] For example, the loaded modes include night scene mode, portrait mode, photo mode, video mode, etc. As shown in FIG4 (B), the electronic device can display the camera application interface 21. As shown in FIG4 (B), the camera application interface 21 can also include a captured image display control 212, a shooting control 213, a camera switching control 214, a viewfinder 215, a focus control 216A, a setting control 216B and a filter control 216C. Among them:

[0109] The shooting control 213 is used to respond to user operations, shoot and save the shot photos.

[0110] The captured image display control 212 is used for the user to view captured pictures and videos.

[0111] The camera switching control 214 is used to switch the camera for collecting images between the front camera and the rear camera.

[0112] The viewfinder 215 is used for real-time preview display of the captured pictures.

[0113] The focus control 216A is used to adjust the focus of the camera.

[0114] The setting control 216B is used to set various parameters when acquiring images.

[0115] The filter control 216C is used to select the filter effect when shooting.

[0116] The camera application interface 21 may include icons 211 corresponding to loaded modes. The icons 211 may include a night mode icon 211A, a portrait mode icon 211B, a photo mode icon 211C, a video mode icon 211D, and more icons 211E. The shooting control module may start the mode corresponding to the icon in response to a user touch operation on any of the icons 211. In an embodiment of the present application, the electronic device may open the camera application in response to a user operation and then display the camera application interface 21 on the display screen. The user may operate any of the above-mentioned mode icons, such as touching to start the corresponding shooting mode, and the electronic device will start the corresponding module in the hardware abstraction layer.

[0117] S103: The user switches to portrait mode.

[0118] 4B , the user may switch to portrait mode by touching the portrait mode icon 211B on the camera application interface 21. In some embodiments, the operation of the user touching the portrait mode icon 211B on the camera application interface 21 to switch to portrait mode may be referred to as a “first user operation”.

[0119] It should be noted in advance that, in some embodiments, the background blur effect of the electronic device on the image in portrait mode can be turned on or off through corresponding operations. Taking Figure 5 as an example, after the shooting mode is switched to portrait mode through the touch operation shown in (B) in Figure 4, the electronic device can display the user interface 31 as shown in (A) in Figure 5. As shown in (A) in Figure 5, the user interface 31 is the shooting interface of the portrait mode in the camera application. The user interface 31 may include a viewfinder 311, a background blur control 312, and a skin beautification control 313. Among them:

[0120] The viewfinder 311 is used to preview the captured image in real time. A face image 3111 and a tree image 3112 are displayed in the viewfinder 311. When the background is blurred, the electronic device can determine the face image 3111 as the foreground (subject) and the tree image 3112 as the background.

[0121] The background blur control 312 is used to blur the background of the image to highlight the foreground (subject) in the image.

[0122] The skin beautification control 313 is used to beautify the appearance of the subject in the image.

[0123] As shown in FIG5(A), when the electronic device just starts the portrait mode, the background blur control can be in the off state by default. Since the electronic device does not blur the background of the image in the preview stream at this time, the electronic device does not need to output the preview image according to the processing logic of subsequent steps S106-S125. After responding to the user's touch operation on the background blur control 312, the electronic device will display the preview image with blurred background in the viewfinder for the user according to the processing logic of subsequent steps S106-S125, and display the user interface 41 shown in FIG5(B).

[0124] From the picture displayed in the preview box 411 in the user interface 41, it can be seen that the overall picture and outline of the facial image 4111 as the foreground are relatively clear, while the background image such as the tree image 4112 has been blurred, so the facial image 4111 is more prominent in the preview box 411.

[0125] Optionally, in some embodiments, when the electronic device just starts the portrait mode, the background blur control can also be turned on by default. That is, after the shooting mode is switched to the portrait mode through the touch operation shown in Figure 5 (B), the electronic device can directly blur the background of the image in the preview stream according to the processing logic of subsequent steps S106-step S125, and the electronic device can directly display the user interface 41 shown in Figure 5 (B).

[0126] S104: The shooting control module is started.

[0127] S105: The preview display module starts.

[0128] Taking (B) in FIG. 4 as an example, the electronic device can activate both the shooting control module and the preview display module in response to the user's touch operation on the portrait mode icon 211B.

[0129] After the capture control module and preview display module are activated, the capture control module may enable the activation of modules related to portrait mode in the hardware abstraction layer, such as the first acquisition module and the second acquisition module. Specifically, the capture control module may issue a preview frame request instruction to the hardware abstraction layer, thereby enabling the activation of the first acquisition module and the second acquisition module in the hardware abstraction layer. The first acquisition module and the second acquisition module may then begin image acquisition.

[0130] In one possible implementation, the shooting control module and the preview display module may have already been activated in step S102, i.e., in response to the user launching the camera application, the shooting control module and the preview display module are activated. The shooting control module can be used for shooting control in various modes. The preview display module can be used for preview display in various modes.

[0131] It should be noted that, in the present application, the shooting control module can continuously issue instructions for controlling the first acquisition module and the second acquisition module to generate RAW images. However, each request instruction generated by the shooting control module will be issued to the first acquisition module, but only one instruction will be issued to the second acquisition module for every two request instructions generated by the shooting control module. Each time a request instruction generated by the shooting control module is received, the acquisition module will generate a frame of RAW image according to the instruction. Therefore, in the present application, the frame rate of the RAW image generated by the first acquisition module is twice the frame rate of the RAW image generated by the second acquisition module.

[0132] Here, all operations performed by the first camera and the second camera under a request instruction issued by the shooting control module are referred to as a working cycle. It can be understood that, since in the embodiment of the present application, the shooting control module will only issue one instruction to the second acquisition module for every two request instructions it generates, the specific working logic of the second acquisition module will also be periodically switched depending on whether a request instruction from the shooting control module is received. In order to introduce the specific working logic of the first camera and the second camera in detail, the specific operating steps of the first camera and the second camera in two consecutive working cycles are described below. Among them, steps S106 to S115 are the first working cycle, steps S107 to S111 are completed by the first camera, and steps S112 to S115 are completed by the second camera; steps S118 to S123 are the second working cycle, steps S119 to S123 are completed by the first camera, and the second camera does not produce frames in the second working cycle.

[0133] S106: The shooting control module sends a request instruction for starting image acquisition to the first acquisition module and the second acquisition module of the hardware abstraction layer.

[0134] S107: The first acquisition module generates a first RAW image.

[0135] S108. The first acquisition module sends the first RAW image to the first IFE module.

[0136] S109: The first IFE module outputs a first RAW image.

[0137] S110 , the first IFE module converts the first RAW image into a first YUV image.

[0138] S111. The first IFE module sends a first YUV image to the IPE module.

[0139] It is understandable that steps S107 to S111 can be performed by the aforementioned first camera (main camera, including the aforementioned first acquisition module and first IFE module) and the IPE module. The first RAW image is generated by the aforementioned first acquisition module and sent to the IPE module.

[0140] Specifically, the first acquisition module can generate the above-mentioned first RAW image according to the preset shooting parameters. The preset shooting parameters may include shooting parameters that may include any one or more of the following: shutter, exposure time, aperture value, exposure value, ISO. The shooting control module can set a shooting parameter and shooting method corresponding to each shooting mode. Exemplarily, the shooting parameter set by the shooting control module in portrait mode may be the first shooting parameter, and the shooting parameter set by the shooting control module in night scene mode may be the second shooting parameter. After the electronic device starts the portrait mode, the shooting control module can send the shooting parameters corresponding to the portrait mode together with the enable startup instruction to the first acquisition module, so that the first acquisition module generates a RAW image according to the shooting parameters corresponding to the portrait mode.

[0141] Specifically, the first acquisition module can acquire the light signal in the shooting scene at the above-mentioned first acquisition rate. After converting the light signal into the above-mentioned first RAW image, the first acquisition module can send the first RAW image to the first IFE module. Since the electronic device needs to use the RAW image stored in the cache to generate photos in the ZSL shooting mode, and the first camera needs to be responsible for outputting the preview image for the user to preview, after the first acquisition module transmits the first RAW image to the first IFE module, the first IFE module will perform two aspects of processing on the first RAW image after receiving the first RAW image: on the one hand, the first IFE module will perform color correction, demosaicing and other processing on the first RAW image, convert the first RAW image into the above-mentioned first YUV image, and transmit the first YUV image to the IPE module, and the IPE module will perform noise reduction, cropping and other processing on the YUV image to obtain the final YUV image for display. On the other hand, the first IFE module will output the RAW image to the cache of the electronic device.

[0142] S112: The second acquisition module generates a second RAW image.

[0143] S113. The second module sends the second RAW image to the second IFE module.

[0144] S114 , the second IFE module outputs the second RAW image to the IPE module.

[0145] S115 . The second IFE module converts the second RAW image into a second YUV image.

[0146] It is understandable that steps S112 to S115 can be performed by the aforementioned second camera (auxiliary camera, including the aforementioned second acquisition module and second IFE module) and the IPE module. The second RAW image is generated by the second acquisition module included in the aforementioned image acquisition module and sent to the IPE module.

[0147] Specifically, the second acquisition module can also generate the second RAW image based on the preset shooting parameters. The shooting control module can set a shooting parameter and shooting method corresponding to each shooting mode. After the electronic device activates portrait mode, the shooting control module can send the shooting parameters corresponding to portrait mode along with an enable instruction to the second acquisition module, so that the second acquisition module generates a RAW image according to the shooting parameters corresponding to portrait mode.

[0148] It should be noted that the first RAW image and the second RAW image may be RAW images generated by the first acquisition module and the second acquisition module at the same time.

[0149] Specifically, the above-mentioned second acquisition module can also collect light signals at a certain acquisition rate, and convert the collected light signals into the above-mentioned second RAW image, and send the second RAW image to the second IFE module. In combination with the above description, it can be seen that in order to reduce power consumption, the electronic device may not use a binocular blur algorithm in the real-time preview stage, but may use a monocular blur algorithm to blur the background of the preview image, and the blurring of the image background does not need to be completed with the help of the YUV image output by the second camera. Therefore, after the second IFE module receives the second RAW image, the second IFE module may not transmit the second YUV image to the IPE module after converting the second RAW image into the above-mentioned second YUV image. However, in order to ensure the blurring effect of the photos taken in the ZSL shooting mode, the electronic device needs to use the binocular blurring algorithm to process the image to obtain the final photo for saving. After the user clicks the shooting control, the electronic device needs to simultaneously obtain the RAW images historically generated by the first camera and the second camera. Therefore, after the second IFE module receives the second RAW image, the second IFE module can convert the second RAW image into the above-mentioned second YUV image, and still output the above-mentioned second RAW image to the cache for use by the electronic device in subsequent shooting.

[0150] In addition, it should be noted that, since steps S107 to S111 and steps S112 to S116 are respectively executed by two different cameras, the electronic device can synchronously execute steps S112 to S115 when executing steps S107 to S111.

[0151] S116 . The IPE module sends a first preview image obtained after processing the first YUV image to the preview display module.

[0152] S117: The preview display module displays a first preview image.

[0153] The first preview image is an image obtained by processing the first YUV image by the electronic device using a monocular blurring algorithm. For details, please refer to the image displayed in the preview box in FIG5 (B).

[0154] S118. The shooting control module sends a request instruction for starting image acquisition to the first acquisition module of the hardware abstraction layer.

[0155] In order to control the frame rate of the RAW image generated by the second camera to be half of that of the first camera, unlike the first working cycle, in the second working cycle, the request instruction generated by the shooting control module will only be sent to the first acquisition module, and will not be sent to the second acquisition module.

[0156] S119: The first acquisition module generates a third RAW image.

[0157] S120: The first acquisition module sends the third RAW image to the first IFE module.

[0158] S121. The first IFE module outputs a third RAW image.

[0159] S122. The first IFE module converts the third RAW image into a third YUV image.

[0160] S123. The first IFE module sends the third YUV image to the IPE module.

[0161] Similarly, steps S120 and S121 may be performed by the aforementioned first camera. The third RAW image is generated by the first acquisition module included in the aforementioned image acquisition module and sent to the IPE module. The third RAW image may be the most recently generated RAW image by the first acquisition module after generating the aforementioned first RAW image.

[0162] Specifically, after generating the first RAW image, the above-mentioned first acquisition module will continue to collect light signals in the scene and convert the light signals into the above-mentioned third RAW image. Therefore, the above-mentioned first acquisition module will continue to send the above-mentioned third RAW image to the first PIE module. Combined with the above description, it can be seen that after receiving the third RAW image, the first IFE module will perform two-way processing on the third RAW image, that is, the first IFE module will convert the third RAW image into the above-mentioned third YUV image, and transmit the third YUV image to the IPE module. In addition, the IPE module will output the RAW image to the cache of the electronic device.

[0163] As can be seen from steps S107-S111 and S119-S123, since each request instruction generated by the capture control module is sent to the first acquisition module, the first camera's operating logic remains the same throughout all operating cycles, generating a single RAW image frame in each cycle. During subsequent capture, the first camera generates and outputs images according to the operations shown in steps S107-S111 (or S119-S123).

[0164] Since the request instruction generated by the shooting control module is not sent to the second acquisition module during the second working cycle, the second acquisition module will not generate and output RAW images in this application. In addition, it can be seen from steps S106 and S118 that since the request instruction generated by the shooting control module is intermittently sent to the second acquisition module, the second camera will generate a frame of RAW image every other working cycle (i.e., every two working cycles). During the subsequent shooting process, the second camera will periodically switch between different working modes to control the frame rate of the RAW image generated by the second acquisition module to be half that of the first acquisition module.

[0165] S124. The IPE module sends a second preview image obtained by processing the third YUV image to the preview display module.

[0166] S125: The preview display module displays the second preview image.

[0167] Similarly, the second preview image is also an image obtained by the electronic device processing the third YUV image using a monocular defocusing algorithm. Specific reference can be made to the image displayed in the preview box in FIG5 (B).

[0168] Afterwards, the first camera and the second camera in the electronic device can continue to output images according to the output logic shown in the above steps. Specifically, the shooting control module will send each request instruction generated to the first acquisition module, and the first acquisition module in the first camera generates a RAW image at a first frame rate. At the same time, the shooting control module sends a request instruction to the second acquisition module every other time (that is, it will send a request instruction to the second acquisition module every time two request instructions are generated), and the second acquisition module in the second camera generates a RAW image at a second frame rate. At the same time, the first acquisition module will transmit the generated RAW image to the above-mentioned first IFE module, and the first IFE module will convert each frame of the received RAW image into a YUV image and send it to the IPE module, and the first IFE module will output each frame of the received RAW image to the cache of the electronic device; the IPE module crops the received YUV image, performs monocular blur processing on the received YUV image, and then outputs it to the screen of the electronic device for user preview. The second acquisition module will also transmit the generated RAW image to the above-mentioned second IFE module. After the second IFE module converts each frame of the received RAW image into a YUV image, it will not send it to the IPE module (or the second IFE module may not convert the RAW image into a YUV image). Accordingly, the IPE module will not output the YUV image if it cannot receive the YUV image transmitted by the IPE module; however, the second IFE module will also output the RAW images received from the second acquisition module to the cache of the electronic device.

[0169] For example, in a third operating cycle following the second operating cycle, the shooting control module will simultaneously send the generated request instruction to both the first acquisition module and the second acquisition module. Consequently, both the first acquisition module and the second acquisition module will each generate a RAW image frame under the control of the instruction. However, in a fourth operating cycle following the third operating cycle, the shooting control module will send the generated request instruction only to the first acquisition module. Consequently, the first acquisition module will generate a RAW image under the control of the instruction, but the second acquisition module will not. In this way, the electronic device can control the second frame rate to be only half the value of the first frame rate, thereby reducing the operating intensity and power consumption of the second camera.

[0170] If the user subsequently clicks on the shooting control, such as the user's touch operation on the shooting control 413 shown in (B) in Figure 5, the electronic device can select several frames of RAW images from the RAW images output to the cache by the first IFE module, and at the same time select the same number of RAW images corresponding to the aforementioned several frames of images from the RAW images output to the cache by the second IFE module, and use the obtained RAW images to perform image background blurring, multi-frame image fusion and other processing, and then use the final frame of image as the captured photo (for details, please refer to the subsequent relevant descriptions of Figures 6-8, which will not be repeated here), and store it in the electronic device.

[0171] Taking Figure 6 as an example, when the user touches the shooting control 413 as shown in Figure 5 (B), the electronic device can display the user interface 51 as shown in Figure 6 (A). As shown in Figure 6 (A), the user interface 51 is the shooting interface of the portrait mode in the camera application. The user interface 51 can include a viewfinder 511, a background blur control 512, and a control 513 for displaying the captured image. Among them:

[0172] The viewfinder 511 will continue to display the preview image to the user in real time. A face image 5111 and a tree image 5112 are displayed in the viewfinder 511. When the background is blurred, the electronic device can determine the face image 5111 as the foreground (subject) and the tree image 5112 as the background. Since the background blur control 512 is already turned on at this time, the tree image 5112 has been blurred. However, since the preview image displayed by the electronic device uses a monocular blur algorithm to blur the image, the blur effect of the preview image may be relatively rough, and it is very likely that the background will be blurred or the subject will be blurred. As shown in (A) in Figure 6, the face image 5111 is the foreground, and its hair part 5111A is also in the foreground, but the hair part 5111A is still blurred.

[0173] The captured image echo control 513 may display a thumbnail of the photo obtained after the user clicks the capture control.

[0174] In response to the user's touch operation on the captured image echo control 513 as shown in (A) in Figure 6, the electronic device can display the user interface 61 as shown in (B) in Figure 6. The user interface 61 can be an application interface of a gallery application, which may include an image 611 for displaying the user's historical photos, that is, the electronic device processes a plurality of RAW images output by the first IFE module and the same number of RAW images output by the second IFE module based on the above-mentioned photos. In the image 611, the face image 6111 is determined as the foreground (subject), and the tree image 6112 is determined as the background. In combination with the signed description, it can be seen that the image 611 is blurred using a binocular blurring algorithm, so the blurring effect of the preview image is more delicate than that of the preview image, wherein the face image 6111 has a clear outline and there is no part that is incorrectly blurred, and the tree image 6112 as the background image is also accurately blurred.

[0175] FIG7 shows a specific image output method when an electronic device implements the shooting method provided in this application in ZSL mode.

[0176] FIG7(A) shows the image outputting mode of the electronic device in the preview stage.

[0177] During the preview phase, the first camera outputs a YUV image for display (such as the first YUV image and the third YUV image mentioned above) at a first frame rate. These YUV images will eventually be drawn on the screen by the electronic device for user preview and display after the electronic device blurs the background based on the monocular blur algorithm. At the same time, the first camera will also generate and output a RAW image (such as the first RAW image and the third RAW image) for conversion into the aforementioned YUV image to the cache of the electronic device at a first frame rate. That is to say, every time the first camera generates a RAW image, the first camera will output the RAW image to the cache of the electronic device; in addition, the first camera will convert the RAW image into a YUV image and then transmit it to the screen of the electronic device (for example, first transmit it to the CPU or GPU, and the CPU or GPU will draw it to the screen of the electronic device) for user preview. Taking image 701 and image 702 in Figure 7 as an example (image 701 and image 702 can be the first YUV image and the first RAW image in the above description respectively), image 702 can be a RAW image generated by the first acquisition module in the first camera, and image 701 can be a YUV image obtained after image 702 is processed by the first IFE module and IPE module in the first camera.

[0178] During the preview phase, the IPE module's YUV image input port (i.e., the data interface used by the IPE module to receive the YUV image from the second IFE module) is always closed under the control of the electronic device. Therefore, the second camera does not output any YUV images for display. Furthermore, under the control of the electronic device, the frame rate at which the first camera generates RAW images is twice that of the second camera. Therefore, for every two RAW frames generated by the first camera, the second camera generates one RAW frame. As shown in Figure 7 (A), when the first camera generates image 702, the second camera also generates image 704 simultaneously. However, when the first camera generates image 703, the second camera does not generate a RAW image. Subsequently, when the first camera generates image 705, the second camera also generates image 706, and so on. Similarly, the second camera also outputs all generated RAW images to the electronic device's buffer for use during the capture phase.

[0179] FIG7(B) shows the image output mode of the electronic device during the shooting stage.

[0180] During the capture phase, the first and second cameras can continue to output images in the same manner as during the preview phase. However, to obtain the captured photo, the electronic device may select and process several frames from the RAW images stored in the buffered memory by the first camera and the RAW images stored in the buffered memory by the second camera to obtain the captured photo. Specifically, the electronic device may determine images 707 and 708 from the RAW images stored in the buffered memory by the first camera, and based on the frame numbers of these two images (which may reflect the generation time of the RAW images; two RAW images with the same frame number have the same generation time), determine images 709 and 710 from the RAW images stored in the buffered memory by the second camera. Images 707 and 708 have the same frame number, and images 709 and 710 have the same frame number. The electronic device may then perform a quality assessment on images 707 and 708, select the image with higher quality (assuming image 708 has higher quality), and select the image with the same ID number as image 708, i.e., image 710, from images 709 and 710. Afterwards, the electronic device can use the binocular blur algorithm to process image 708 and image 710 to obtain image 711 with a blur effect. Correspondingly, the electronic device can use the binocular blur algorithm to process image 707 and image 709 to obtain image 711 with a blur effect, and store the final image 711 as the above-mentioned photo in the electronic device (for example, in a gallery).

[0181] Of course, in some embodiments, the electronic device may also select a larger number of images (for example, three frames of images each) from the RAW images output from the first camera to the cache and the RAW images output from the second camera to the cache, and obtain the above-mentioned photos according to the above-mentioned processing method. This application does not limit this.

[0182] In addition, in some embodiments, the electronic device may also use a binocular blur algorithm and a multi-frame fusion algorithm to obtain the above-mentioned first preview image and second preview image. For example, the electronic device may first determine the key frame and the reference frame in the RAW image output by the first camera, first fuse the multiple frames of RAW image output by the first camera to obtain a fused image, and then calculate the depth of field information of the image based on the reference frame and the RAW image corresponding to the reference frame (that is, the RAW image output by the second camera at the same time as the reference frame generation time), and blur the background of the above-mentioned fused image based on the depth of field information, and store the blurred image as the above-mentioned photo in the electronic device (for details, please refer to the above-mentioned relevant description of Figure 1, which will not be repeated here).

[0183] Optionally, after the user clicks the shooting control, the electronic device may use other methods to process the historically cached RAW images to obtain the above-mentioned photos. The specific method may be determined by the algorithm logic of the multi-frame fusion algorithm and binocular blur algorithm used by the electronic device, and this application does not limit this. However, in order to ensure the background blur effect of the photo and the absence of a delay between the photo image and the moment the user took the photo, the historically cached RAW image obtained by the electronic device (i.e., the RAW image stored before the user clicked the shooting control) must contain both the RAW image historically output by the first camera and the RAW image historically output by the second camera.

[0184] In addition, in combination with the above description, it can be seen that the RAW images output by the above-mentioned first IFE module and the above-mentioned second IFE module will be saved in the cache (hereinafter, the RAW images output by the first IFE module and saved in the cache will be referred to as the RAW images output by the main camera, and the RAW images output by the second IFE module and saved in the cache will be referred to as the RAW images output by the auxiliary camera). In addition, in order to be able to determine a pair of RAW images with the same generation time from the RAW images output by the two cameras in the subsequent shooting and frame selection process, when outputting the RAW images to the cache, the electronic device will also add frame numbers to the RAW images, wherein the RAW images generated at the same time (such as the above-mentioned first RAW image and second RAW images) have the same frame numbers. When selecting the RAW images, the electronic device can determine the required RAW images and the frame numbers corresponding to these RAW images from the RAW images output by one of the cameras, and then select the RAW images with the same frame numbers from the RAW images output by the other camera based on the frame numbers corresponding to these RAW images.

[0185] However, due to storage space limitations and the fact that the second IFE stores RAW images at intervals, when the user subsequently clicks the shooting control, when the electronic device selects corresponding RAW images from the RAW images output by the main camera and the RAW images output by the auxiliary camera, after the electronic device determines multiple frames of RAW images from the RAW image output by the main camera, the electronic device may not be able to completely find the RAW images with the same frame number of the multiple frames of RAW images in the RAW images output by the other camera.

[0186] For illustration, let's assume that the electronic device stores 50 RAW images generated by both the first and second cameras, each with frames numbered 1-50. Furthermore, the first cache queue for storing the RAW images output by the first camera can store a maximum of 10 RAW images, and the second cache queue for storing the RAW images output by the second camera can also store a maximum of 10 RAW images. When a new RAW image needs to be cached in the queue, the oldest RAW image in the queue is cleared. As shown in (A) of FIG8 , the first cache queue contains the 10 most recent RAW images output by the first camera, with frames numbered 41-50. The second cache queue also contains the 10 most recent RAW images output by the second camera. However, since the second camera generates RAW images at only half the frame rate of the first camera, the second camera also outputs RAW images at only half the frame rate of the first camera. Therefore, the frame numbers corresponding to the 10 RAW images cached in the second cache queue are 32, 34, 36, 38, 40, 42, 44, 46, 48, and 50 respectively. After the user clicks the shooting control, assuming that the electronic device needs to select 3 frames of images from the first cache queue and the second cache queue respectively, and obtain the above-mentioned photo according to the processing logic described above, and the electronic device selects three RAW images with frame numbers 46, 47, and 48 from the first cache queue, then the electronic device needs to select three RAW images with the same frame number from the second cache queue based on the frame numbers of these three RAW images. However, as can be seen from Figure 8, there are only RAW images with frame numbers 46 and 48 in the second cache queue, but there is actually no RAW image with frame number 47. Therefore, the electronic device may not be able to output the above-mentioned photo normally.

[0187] To address the above shortcomings, in one possible implementation, the electronic device can halve the length of the second cache queue, retaining only the five most recently stored RAW image frames. This ensures that for every RAW image in the second cache queue, there is a RAW image with the same frame number in the first cache queue. The electronic device can then determine the frame numbers of the three required RAW images from the second cache queue, and then find the other three RAW images corresponding to these three RAW image numbers in the first cache queue.

[0188] As shown in (B) of FIG8 , after the length of the cache in the second cache queue is halved, the frame numbers corresponding to the 10 most recently cached RAW images in the first cache queue remain 41-50, and the frame numbers corresponding to the 5 most recently cached RAW images in the second cache queue are 42, 44, 46, 48, and 50. It is understandable that the set {42, 44, 46, 48, 50} is included in the set {41, 42, 43, 44, 45, 46, 47, 48, 49, 50}. Therefore, after the user clicks the capture control, no matter what frame numbers the electronic device determines the required RAW images to be from the second cache queue, the electronic device can select RAW images with the same frame numbers from the first cache queue based on these frame numbers. For example, assuming that the electronic device determines from the second cache queue that the frame numbers of the three required RAW images are 44, 46, and 48, the electronic device can select the three RAW images with frame numbers 44, 46, and 48 from the first cache queue, and obtain the above-mentioned photos through these 6 RAW images.

[0189] It is understood that the embodiments of the present application are described using image capture in portrait mode as an example. However, the embodiments of the present application are not limited to portrait mode and can also use the above-mentioned capture method in other capture modes involving multiple cameras. The embodiments of the present application are not limited to this. In addition, in some embodiments, the frame rate of the RAW image generated by the second camera can also be 1 / 3, 1 / 4, or other values ​​of the frame rate of the RAW image generated by the first camera, as long as the frame rate of the RAW image generated by the second camera is lower than the frame rate of the RAW image generated by the first camera. This is not limited to this in the present application.

[0190] It should be understood that in order to ensure the quality of the photos taken by the user, the time when the second camera generates the RAW image needs to be as consistent as possible with the time when the first camera generates the RAW image in the same working cycle. In particular, when there are moving targets in the shooting scene, the depth of field information provided by the images selected from the two cache queues will only be accurate if the image output time is the same. Only then can the electronic device better identify the foreground and background of the image and blur the image more accurately.

[0191] However, in some embodiments, due to performance limitations of the first and second cameras, although the electronic device may maintain a RAW image frame rate of 1 / 2 that of the first camera, the time it takes for the second camera to generate a RAW image may differ significantly from the time it takes for the first camera to generate a RAW image within the same operating cycle. For example, when a user launches a camera application and enters portrait mode, the first and second cameras, due to performance differences, may not be activated at the same time and therefore may not begin collecting light signals and generating RAW images at the same time. This may result in a significant difference between the time it takes for the second camera to generate a RAW image and the time it takes for the first camera to generate a RAW image in all subsequent operating cycles. Alternatively, due to performance or other reasons, the first and / or second cameras may not consistently maintain a consistent frame interval between generated images when generating RAW images. Consequently, the frame interval between a particular RAW image generated at a given moment may be significantly different from the frame interval between other RAW images. This may also result in a significant difference between the time it takes for the second camera to generate a RAW image and the time it takes for the first camera to generate a RAW image in all subsequent operating cycles.

[0192] To address the above issues, in some embodiments, the electronic device can periodically select two frames with the same frame number from the primary and secondary RAW image buffer queues, and use the timestamp information of these two frames to determine whether the first and second cameras generated these two frames at the same time. Because the frame rate of the RAW images output by the first and second cameras is fixed, the frame interval between the RAW images output by the first and second cameras is also relatively stable (without significant fluctuation). Therefore, if there is a large time interval (e.g., greater than or equal to 1ms) between the times when the first and second cameras generate these two frames, the electronic device can adaptively adjust the frame interval between the latest RAW image generated by the image sensor of the second camera and the previous RAW image before the image sensor of the second camera generates the latest RAW image. In other words, based on the time interval, the electronic device can adaptively advance or delay the generation of the latest RAW image that was originally due at a certain time, so that the image sensors of the second camera and the first camera can each generate the latest RAW image at the same time. For details, please refer to Figure 9.

[0193] Figure 9 (A) shows a scenario where the second camera generates RAW images at a different time than the first camera generates RAW images. As shown in Figure 9 (A), images 901 and 902, images 903 and 904, images 905 and 906, and images 907 and 908 are RAW images generated by the first and second cameras during four different working periods. It is assumed here that images 901 and 902 are RAW images generated by the first and second cameras, respectively, during the first cycle. Due to the performance differences between the two cameras, there is a time interval of duration T0 between the time t91 when the first camera generates image 901 and the time t90 when the second camera generates image 902. This means that during the first cycle, the first camera generates the RAW image T0 later than the second camera (T0 duration is greater than or equal to 1ms). It is understandable that, without any intervention, in the third working cycle (the working cycle for generating images 903 and 904), the fifth working cycle (the working cycle for generating images 905 and 906), and the seventh working cycle (the working cycle for generating images 907 and 908), there is as long a time interval as possible between the time when the first camera generates a RAW image and the time when the second camera generates a RAW image. Here, it is assumed that these are T1, T2, and T3, respectively. It is understandable that, if the frame intervals of the RAW images generated by the first camera and the second camera are subsequently stable, the T0, T1, T2, and T3 durations may be the same. If the frame intervals of the RAW images generated by the first camera or the second camera subsequently experience one or more abnormal fluctuations, the T0, T1, T2, and T3 durations may be different.

[0194] As can be seen from (A) in Figure 9, the four groups of images, namely image 901 and image 902, image 903 and image 904, image 905 and image 906, and image 907 and image 908, are not RAW images generated at the same time. In addition, without intervention, in the subsequent working cycle, there may always be a large time interval between the time when the first camera and the second camera generate RAW images. Therefore, in the subsequent shooting process, the primary and auxiliary RAW images selected by the electronic device are not the images obtained by the first camera and the second camera shooting the scene at the same time, which is likely to result in poor quality of the final photo image.

[0195] (B) in Figure 9 shows a scenario in which the electronic device adjusts the time when the second camera generates a RAW image in the fifth working cycle (the working cycle for generating images 905 and 906), so that the moment when the second camera subsequently generates the RAW image is as close as possible to the moment when the first camera generates the RAW image.

[0196] It should be noted in advance that, since the electronic device needs to use the timestamp information of the two frames of images output by the primary and secondary cameras stored in history to determine the time interval between the times when the first camera and the second camera respectively generate RAW images, and adjust the time when the image sensor in the second camera generates the latest frame of RAW image based on this interval. Therefore, the two frames of images output by the primary and secondary cameras obtained by the electronic device need to carry corresponding timestamp information. However, whether it is the first camera or the second camera, the RAW image generated by it will only carry timestamp information after it is output to the cache outside the image sensor (such as the aforementioned main RAW image cache queue and the auxiliary RAW image cache queue). Usually, before the latest frame of RAW image is generated, the image sensor generally has not had time to output the latest 2-3 frames of images generated historically to the cache outside the image sensor, so these images will not carry corresponding timestamp information. Taking (A) in Figure 9 as an example, at time t94, the image sensor in the second camera failed to output image 904, and at time t95, the image sensor in the first camera also failed to output image 903. Therefore, neither image 903 nor image 904 carries timestamp information. In other words, if the electronic device needs to adjust the time when the second sensor generates image 906 so that the time interval between the time when the second camera generates the RAW image and the time when the first camera generates the RAW image is shortened, the electronic device can only obtain RAW images that were generated and output to the cache queue by the first and second cameras at earlier times and carry timestamp information, such as images 901 and 902. Based on the timestamp information of these two frames, the electronic device determines the approximate time interval between the time when the second camera currently generates the RAW image and the time when the first camera generates the RAW image, and adjusts the time when the image sensor in the second camera generates the latest RAW image based on this time interval.

[0197] That is, in this embodiment of the present application, the electronic device can obtain the first N1 RAW image frames generated by the first camera and the first N2 RAW image frames generated by the second camera to determine the approximate time interval between the time when the second camera generated the RAW image and the time when the first camera generated the RAW image. Specifically, the value of N1 can be 4, and the value of N2 can be 2.

[0198] As shown in (B) of FIG9 , during the fifth operating cycle, the electronic device can obtain images 901 and 902, which are generated by the first and second cameras and output to the cache queue and carry timestamp information. Based on the timestamp information of these two frames, the electronic device can determine that the time when the second camera generated image 902 is time t90, and the time when the first camera generated image 901 is time t91. The electronic device can then determine that the time when the second camera generated the RAW image during the same operating cycle was earlier than the time when the first camera generated the RAW image by a time duration T0. Therefore, during the fifth operating cycle, according to the frame interval specified by the second camera for generating the RAW image and the time when it generated the previous RAW image (i.e., image 904) (i.e., time t92), the second camera should have generated image 906 at time t94 after time t92 under the control of the electronic device. However, due to the intervention of the electronic device, the second camera actually generated image 906 at time t94', which is a time duration T0 after time t94, and the electronic device did not change the time when the first camera generated image 905. It is understandable that under this intervention, in the fifth working cycle, the time when the second camera generates the RAW image is T0 later than before, and the time interval between the time when the second camera generates the RAW image and the time when the first camera generates the RAW image is also shortened from the original T2 to (T2-T0), and the time when the two cameras generate the images can be closer; if T2=T0, then there may not even be a time interval between the time when the second camera generates the RAW image and the time when the first camera generates the RAW image. In this case, the image output times corresponding to the images subsequently selected by the electronic device from the two cache queues are closer, and the depth of field information obtained based on the two images is more accurate, so the electronic device can blur the image more accurately.

[0199] Optionally, in the subsequent image output process, in order to save the computing power of the electronic device and reduce the power consumption of the device, the electronic device can periodically obtain the first N1 frame RAW image generated by the first camera and the first N2 frame RAW image generated by the second camera, with each cycle of 5 frames of image generated by the first camera, and determine the approximate time interval between the time when the second camera currently generates the RAW image and the time when the first camera generates the RAW image, and adjust the time when the second camera generates the latest RAW image based on the time interval, so that the time when the second camera subsequently generates the RAW image is as close as possible to the time when the first camera generates the RAW image. For example, after adjusting the time when the second camera generates image 906 based on the timestamp information of images 901 and 902 in the fifth working cycle, the electronic device may no longer intervene in the time when the camera generates the RAW image in the subsequent five working cycles (including the three cycles in which the second camera does not need to generate RAW images). Not until the eleventh operating cycle does the electronic device obtain images 907 and 908, which are generated by the first and second cameras and output to the cache queue and carry timestamp information. Based on the timestamps of these two images, the electronic device determines that the time when the second camera generated image 908 was t96' (due to the intervention of the fifth cycle on the time when the second camera generated the RAW image, the time when the second camera generated image 908 was delayed by T0 from the original time t96 to t96' during the seventh cycle). Furthermore, the time when the first camera generated image 903 was t97. Therefore, the electronic device can determine that the time when the second camera generated the RAW image during the same operating cycle was earlier than the time when the first camera generated the RAW image by (T3-T0) (here, assuming that (T3-T0) is greater than 1ms). Therefore, during the eleventh operating cycle, the electronic device's intervention delayed the time when the second camera generated the latest RAW image by (T3-T0), while the electronic device did not change the time when the first camera generated the latest RAW image. And so on.

[0200] Alternatively, during subsequent image output, before the second camera generates the latest RAW image frame, the electronic device may obtain the N1th RAW image frame generated by the first camera and the N2th RAW image frame generated by the second camera, determine the approximate time interval between the time when the second camera generates the RAW image and the time when the first camera generates the RAW image, and adjust the time when the second camera generates the latest RAW image frame based on this time interval so that the time when the second camera generates the RAW image is as close as possible to the time when the first camera generates the RAW image. For example, in the seventh working cycle, before outputting image 908, the electronic device may obtain images 903 and 904, which are generated by the first camera and the second camera and output to the cache queue and carry timestamp information, and determine based on the timestamp information of these two images that the time when the second camera generated image 904 was time t92 and the time when the first camera generated image 903 was time t93. The electronic device can then determine that the time when the second camera generated the RAW image in the same working cycle was earlier than the time when the first camera generated the RAW image by a duration of T1. Therefore, in the seventh working cycle, the second camera should have generated image 906 at time t96. However, due to the intervention of the electronic device, the second camera generated image 906 at time t94 (not shown in Figure 9), which is after time t94 and a time T1 away from time t96. The electronic device will not change the time when the first camera generated image 905, and so on.

[0201] Figure 10 shows the process of an electronic device executing the above-mentioned shooting method from the perspective of the Android system architecture. As shown in Figure 10, the architecture involved in the electronic device executing the above-mentioned shooting method may include the following three layers: application layer, application framework layer, and hardware abstraction layer. The specific functions of each layer of architecture in the Android system can be referred to the above-mentioned related descriptions, which will not be repeated here. In the process of the electronic device executing the above-mentioned shooting method, the application layer, application framework layer, and hardware abstraction layer can work together to complete the entire preview process, specifically:

[0202] The application layer can respond to user operations, such as a user opening a camera application, by continuously sending request instructions to the application framework layer. Each frame request instruction corresponds to a preview image displayed on the electronic device screen. Alternatively, the application layer can respond to user operations, such as a user clicking a camera control, by continuously sending capture and image output request instructions to the application framework layer. The capture and image output request instructions can be used to instruct the hardware abstraction layer to use the cached RAW image to obtain a photo.

[0203] Afterwards, the application framework layer will further send the request instructions or image capture request instructions transmitted from the application layer to the hardware abstraction layer, and the hardware module in the hardware abstraction layer will output the corresponding RAW image and / or YUV image according to the instructions. For details, please refer to the aforementioned relevant descriptions of Figures 2 and 3, which will not be repeated here.

[0204] Afterwards, the hardware layer can pass the generated preview image (the preview image can be some drawing instructions at this time) back to the application layer through the application framework layer, and SurfaceFlinger will draw the preview image on the screen of the electronic device for the user to browse according to the drawing instructions.

[0205] Next, the electronic device provided by the embodiments of the present application is introduced.

[0206] The electronic device may be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), or a dedicated camera (e.g., a single-lens reflex camera or a compact camera). The present application does not impose any restrictions on the specific type of the electronic device. Specifically, the electronic device 100 may be the electronic device described above.

[0207] FIG11 exemplarily shows the structure of the electronic device.

[0208] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, a button 190, a motor 191, a camera 193, a display screen 194, etc.

[0209] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0210] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0211] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0212] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0213] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0214] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C busses. The processor 110 can be coupled to a charger, flash, camera 193, etc. through different I2C bus interfaces.

[0215] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the electronic device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0216] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0217] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0218] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0219] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, etc. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be provided in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be provided in the same device.

[0220] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker, a receiver, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module or other functional modules.

[0221] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0222] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0223] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0224] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise and brightness. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0225] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include N cameras 193, where N is a positive integer greater than 1.

[0226] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0227] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0228] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0229] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0230] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0231] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0232] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0233] In this application, the camera 193 includes at least two cameras, including a first camera and a second camera, both of which are front cameras or rear cameras of the electronic device. Optionally, the first camera can be a wide-angle camera and the second camera can be an ultra-wide-angle camera; or the first camera can be a telephoto camera and the second camera can be a wide-angle camera.

[0234] In an embodiment of the present application, the processor 110 can control the frame output mode of the first camera and the second camera in the ZSL mode.

[0235] During the preview phase of the ZSL shooting mode, the first ISP (including the first IFE) in the first camera may convert the RAW image generated by the first sensor in the first camera into a YUV image and output it. The second ISP (including the second IFE) in the second camera may not output the RAW image generated by the second sensor in the second camera. The processor 110 may process the YUV image output by the first camera based on a monocular defocusing algorithm and then, in conjunction with the GPU, render the processed image to the display screen 194 for display.

[0236] Furthermore, during the preview phase of ZSL shooting mode, the first camera generates and outputs RAW images at a first frame rate, while the second camera generates and outputs RAW images at a second frame rate, where the first frame rate is greater than the second frame rate. Optionally, the first frame rate is twice the second frame rate, meaning that for every two RAW frames generated by the first image sensor in the first camera, only one RAW frame is generated by the second image sensor in the second camera.

[0237] The RAW images output by the first camera and the second camera can be cached in a cache area of ​​the electronic device. The cache area can be a storage area in the memory included in the processor 110, a storage area in an external memory, or a storage area in the internal memory 121.

[0238] During the shooting stage in the ZSL shooting mode, in response to the user's touch operation on the shooting control, the processor 110 can select RAW images with the same number of frames from the RAW images output by the first camera and the RAW images output by the second camera, respectively, to obtain M groups of RAW images with the same generation time (i.e., two RAW images with the same frame number). The processor 110 can use the binocular blur algorithm to process each group of RAW images to obtain M frame images (the backgrounds of these images have a blur effect), and use a multi-frame fusion algorithm to fuse the M frame images to obtain the final photos that need to be stored.

[0239] Optionally, after the user clicks the shooting control, the electronic device may use other methods to process the historically cached RAW images to obtain the above-mentioned photos. The specific method may be determined by the algorithm logic of the multi-frame fusion algorithm and binocular blur algorithm used by the electronic device, and this application does not limit this. However, in order to ensure the background blur effect of the photo and the absence of a delay between the photo image and the moment the user took the photo, the historically cached RAW image obtained by the electronic device (i.e., the RAW image stored before the user clicked the shooting control) must contain both the RAW image historically output by the first camera and the RAW image historically output by the second camera.

[0240] An embodiment of the present application further provides an electronic device, the electronic device comprising: one or more processors and a memory;

[0241] The memory is coupled to the one or more processors, and the memory is used to store computer program code, which includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method shown in the above embodiment.

[0242] As used in the above embodiments, the term “when…” may be interpreted to mean “if…” or “after…” or “in response to determining…” or “in response to detecting…”, depending on the context. Similarly, the phrases “upon determining…” or “if (stated condition or event) is detected” may be interpreted to mean “if determining…” or “in response to determining…” or “upon detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.

[0243] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0244] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A shooting method, characterized in that, The method includes: In response to a first user operation, starting a portrait shooting mode, where the portrait shooting mode is a mode of jointly shooting with a first camera and a second camera; Controlling the first camera to generate a RAW image at a first frame rate, and controlling the second camera to generate a RAW image at a second frame rate, where the first frame rate is greater than the second frame rate.

2. The method according to claim 1, wherein The first frame rate is 2 times the second frame rate. The controlling the first camera to generate a RAW image at a first frame rate and controlling the second camera to generate a RAW image at a second frame rate includes: At a first moment, controlling the first camera to generate a first RAW image and controlling the second camera to generate a second RAW image; At a second moment, controlling the first camera to generate a third RAW image.

3. The method according to claim 2, wherein Before the controlling the first camera to generate a first RAW image and controlling the second camera to generate a second RAW image at the first moment, the method further includes: Determining a first duration based on a fourth RAW image generated by the first camera and a fifth RAW image generated by the second camera, where the frame numbers of the fourth RAW image and the fifth RAW image are the same, and the first duration is the time interval between the moment when the first camera generates the fourth RAW image and the moment when the second camera generates the fifth RAW image; Determining the first moment based on the first duration and the second frame rate.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Controlling the first camera to output a YUV image at the first frame rate, Processing the first YUV image output by the first camera based on a monocular blurring algorithm to obtain a first preview image; Displaying the first preview image in a preview area of a display screen.

5. The method according to any one of claims 1 to 4, characterized in that, The method is applied to a zero-second delay shooting mode, and the method further includes: In response to a user operation on a shooting control, saving a first photo, where the first photo is obtained by processing at least one frame of the first RAW image generated and cached by the first camera and at least one frame of the second RAW image generated and cached by the second camera based on a binocular blurring algorithm.

6. The method according to any one of claims 1 to 5, characterized in that The first frame rate is n times the second frame rate. The RAW images generated by the first camera are output to a first cache queue, and the RAW images generated by the second camera are output to a second cache queue. In the case where both the first cache queue and the second cache queue are full, the number of RAW images stored in the second cache queue is n times the number of RAW images in the first cache queue, and n is a positive number.

7. The method according to claim 5 or 6, characterized in that, Before saving the first photo, the method further includes: Determining at least one frame of second RAW image from the RAW images output by the second camera; Determining at least one frame of first RAW image from the RAW images output by the first camera based on the frame numbers of the at least two frames of RAW images, where the number of the at least one frame of first RAW image is the same as the number of the at least one frame of second RAW image; Fusing the at least one frame of first RAW image based on a multi-frame fusion algorithm to obtain a first image; Process the first key frame and the second key frame based on the binocular defocusing algorithm to obtain the depth-of-field information of the first key frame. The first key frame is a RAW image with better image quality among the at least one first RAW image, and the second key frame is a RAW image with the same frame number as the first key frame among the at least one second RAW image; Blur the background of the first image based on the depth-of-field information to obtain the first photo.

8. The method according to any one of claims 1 to 7, wherein The first camera may be a wide-angle camera, and the second camera may be an ultra-wide-angle camera; Alternatively, the first camera is a telephoto camera, and the second camera may be a wide-angle camera.

9. An electronic device, characterized in that, The electronic device includes: one or more processors, a memory, and a display screen; The memory is coupled to the one or more processors. The memory is used to store computer program code, and the computer program code includes computer instructions. The one or more processors call the computer instructions to cause the electronic device to execute the method according to any one of claims 1-8.

10. A chip system, characterized in that, The chip system is applied to an electronic device. The chip system includes one or more processors, and the processors are used to call computer instructions to cause the electronic device to execute the method according to any one of claims 1-8.

11. A computer-readable storage medium, comprising instructions, characterized in that, When the instructions run on the electronic device, the electronic device is caused to execute the method according to any one of claims 1-8.

Images