Shooting method and electronic equipment

Abstract

The invention provides a shooting method and electronic equipment, and the shooting method reduces the working intensity of an auxiliary road camera through reducing the picture output frame rate of the auxiliary road camera in a preview stage under the condition of maintaining the picture output frame rate of a main road camera, reduces the power consumption of the electronic equipment during shooting under the premise of maintaining the quality of a shot picture, and improves the shooting efficiency of the electronic equipment. And the computing power burden on the electronic equipment is reduced.

Description

Technical Field

[0001] This application relates to the technical field of terminals, and in particular, to a shooting method and an electronic device. Background Art

[0002] Nowadays, the shooting and beautification of images have become an indispensable part of our daily life and entertainment. Most electronic devices currently use two cameras to take pictures when shooting, so as to create various image effects. For example, one of the cameras can be responsible for shooting the overall picture. Another camera can be used to obtain depth-of-field information to blur the background outside the main subject in the image, thereby highlighting the main subject in the image. However, the operation of multiple cameras together will increase the power consumption of the electronic device, and also maximize the occupation of the memory resources and computing power of the electronic device.

[0003] Therefore, how to reduce the power consumption of multi-camera devices during the shooting process is an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this application is to provide a shooting method and an electronic device. Under the condition of maintaining the output frame rate of the main camera, the shooting method reduces the working intensity of the auxiliary camera by reducing the output frame rate of the auxiliary camera in the preview stage, and reduces the power consumption of the electronic device during shooting and the burden on the computing power of the electronic device on the premise of maintaining the quality of the taken photos.

[0005] The above objectives and other objectives will be achieved by the features in the independent claims. Further implementation manners are reflected in the dependent claims, the description and the drawings.

[0006] In a first aspect, this application provides a shooting method, and the method includes: in response to a first user operation, starting a portrait shooting mode, where the portrait shooting mode is a mode of jointly shooting using a first camera and a second camera; controlling the first camera to generate RAW images at a first frame rate, and controlling the second camera to generate RAW images at a second frame rate, where the first frame rate is greater than the second frame rate.

[0007] In this application, in order for the electronic device to create a specific image effect for the image using a binocular algorithm (such as a binocular blurring algorithm) when the user shoots and outputs the image later, the electronic device needs to be able to obtain the RAW images stored in the history of both cameras at the same time after the user clicks to shoot. Therefore, in this application, the first camera and the second camera can generate and output RAW images simultaneously in the preview stage, and the output RAW images will be output to the cache.

[0008] In this method, the RAW image generated by the first camera can be processed and then used for display for the user to preview. The RAW image generated by the second camera can be used as auxiliary information to add a specific image effect to the displayed image. To reduce the power consumption of the electronic device in the shooting scenario while ensuring the image quality of the photo, the electronic device can control the frame rate of the first camera to generate RAW images to be a relatively large frame rate, that is, the first frame rate, and control the frame rate of the second camera to generate RAW images to be a relatively small frame rate, that is, the second frame rate. In this way, when taking pictures in the subsequent process, the electronic device can not only select images with better image quality from a sufficient number of RAW images previously output by the first camera, but also select images from the RAW images previously output by the second camera for use in assisting the implementation of the binocular algorithm to create an image effect (for example, using the binocular blurring algorithm to blur the background of the image), so as to reduce the power consumption of the electronic device in the shooting scenario while ensuring the image quality of the photo.

[0009] In an alternative embodiment, the first user operation may be an operation in which the user clicks the portrait mode option after opening the shooting software (such as a camera) to cause the electronic device to switch to the portrait shooting mode. After entering the portrait shooting mode, the electronic device can continuously issue instructions for controlling the first camera and the second camera to generate RAW images. However, each request instruction generated by the electronic device will be issued to the first camera, but only one instruction will be issued to the second camera for every several request instructions generated by the electronic device. For each request instruction received from the electronic device, the camera will generate one frame of RAW image according to the instruction. Therefore, in this application, the frame rate of the first camera to generate RAW images is higher than that of the second camera to generate RAW images.

[0010] In combination with the first aspect, in a possible embodiment, the first frame rate is twice the second frame rate. The control of the first camera to generate RAW images at the first frame rate and the control of the second camera to generate RAW images at the second frame rate include: at the first moment, controlling the first camera to generate a first RAW image and controlling the second camera to generate a second RAW image; at the second moment, controlling the first camera to generate a third RAW image.

[0011] In this embodiment, the first moment and the second moment respectively correspond to two consecutive working cycles of the camera, and at the second moment, the second camera does not output a RAW image. That is to say, for every two frames of RAW images generated and output by the image sensor in the first camera under the control of the electronic device, the image sensor in the second camera will generate one frame of RAW image under the control of the electronic device. It can be understood that although the second camera can also be spaced by more frames (for example, the second camera generates one frame of RAW image only when the first camera generates 3 or 4 frames), thereby reducing the power consumption of the second camera on the electronic device to a greater extent. However, when the second camera generates one frame of RAW image when the first camera generates 3 or 4 frames, it is more likely to select a RAW image closer to the user's shooting time when outputting images later. Then, the image content of the photo finally obtained based on the historical cached RAW images will also be more in line with the content expected by the user for shooting, and the image quality can be maximally guaranteed while reducing the power consumption of the second camera on the electronic device.

[0012] Combined with the first aspect, in a possible implementation, before controlling the first camera to generate the first RAW image and controlling the second camera to generate the second RAW image at the first moment, the method further includes: determining a first duration based on the fourth RAW image generated by the first camera and the fifth RAW image generated by the second camera, where the frame numbers of the fourth RAW image and the fifth RAW image are the same, and the first duration is the time interval between the moment when the first camera generates the fourth RAW image and the moment when the second camera generates the fifth RAW image; determining the first moment based on the first duration and the second frame rate.

[0013] However, in some embodiments, due to the limitations in performance of the first camera and the second camera, although the electronic device can maintain the frame rate of the RAW images generated by the second camera at 1 / 2 of the frame rate of the RAW images generated by the first camera, there may be a relatively large gap between the time when the second camera generates RAW images and the time when the first camera generates RAW images in the same working cycle. Therefore, in this embodiment, in order to ensure that in the same working cycle, the time when the second camera generates RAW images is as close as possible to the time when the first camera generates RAW images, so that the electronic device can better identify the foreground and background of the image and more accurately blur the image, the electronic device can periodically select two images with the same frame number from the main and auxiliary RAW image buffer queues respectively, that is, the fourth RAW image and the fifth RAW image, and determine the time interval, that is, the first duration, between the moments when the first camera and the second camera respectively generate these two images through the timestamp information of these two images. Since the frame rates of the RAW images output by the first camera and the second camera are determined, the frame intervals between the RAW images output by the first camera and the second camera are also relatively stable (in the case of no obvious fluctuations). If the first duration is greater than a preset threshold (such as greater than or equal to 1 ms), the electronic device can adaptively adjust the frame interval between the image sensor in the second camera generating the second RAW image and the previous RAW image before the second camera generates the latest frame of RAW image (that is, the second RAW image), that is, according to the above first duration and the specified frame interval, adaptively advance or delay the generation of the second RAW image that should have been generated at a certain moment, so that the image sensors in the second camera and the first camera can generate the latest frame of RAW images at the same moment respectively.

[0014] Combined with the first aspect, in a possible implementation manner, the method further includes: controlling the first camera to output YUV images at the first frame rate, processing the first YUV images output by the first camera based on a monocular blurring algorithm to obtain a first preview image; and displaying the first preview image in a preview area of the display screen.

[0015] It can be understood that although the virtualization effect presented by the monocular virtualization algorithm is not as accurate as that presented by the binocular virtualization algorithm, the monocular virtualization algorithm has lower requirements for the computing power of the device and lower power consumption than the binocular virtualization algorithm. Moreover, what the user ultimately needs to obtain is the captured photo rather than the preview image displayed during preview. Therefore, in this embodiment, the second camera may not output a YUV image during the preview stage. That is, the second camera may still output a RAW image to the cache of the electronic device, but the second camera will not output the converted YUV image from the RAW image. The images sent for display in the preview stream are all processed based on the monocular virtualization effect on the YUV image output by the first camera, and the power consumption of the electronic device can be reduced by sacrificing the quality of the virtualization effect of the images in the preview stream.

[0016] In combination with the first aspect, in a possible implementation, the method is applied to a zero-second delay shooting mode, and the method further includes: in response to a user's operation on a shooting control, saving a first photo, where the first photo is processed based on a binocular virtualization algorithm on at least one frame of first RAW image generated and cached by the first camera and at least one frame of second RAW image generated and cached by the second camera.

[0017] In this method, although the output frame rate of the second camera for generating RAW images is reduced, the image front end in the second camera will still output and cache each frame of RAW image generated and transmitted by the image sensor in the cache queue. Therefore, in this embodiment, after the user clicks the shooting control to take a photo subsequently, the electronic device can still obtain the RAW image cached by the main camera (i.e., the first camera) and the RAW image cached by the auxiliary camera (i.e., the second camera) from the cache, and use the binocular virtualization algorithm to virtualize the two images to ensure the virtualization effect of the finally saved first photo.

[0018] In combination with the first aspect, in a possible implementation, the first frame rate is n times the second frame rate. The RAW images generated by the first camera are output to a first cache queue, and the RAW images generated by the second camera are output to a second cache queue. When both the first cache queue and the second cache queue are full, the number of RAW images stored in the second cache queue is n times the number of RAW images in the first cache queue, and n is a positive number.

[0019] In order to determine a pair of RAW images with the same generation time from the two-way RAW images output by the first camera and the second camera during the subsequent frame selection for shooting, when outputting the RAW images to the buffer, the electronic device adds frame numbers to the RAW images. The RAW images generated at the same moment have the same corresponding frame numbers. When selecting RAW images, the electronic device can determine the required RAW images and the corresponding frame numbers from the RAW images output by the first camera, and then select the RAW images with the same frame numbers from the RAW images output by the other camera based on the frame numbers corresponding to these RAW images. However, due to storage space limitations and the reason that the second camera outputs RAW images at intermittent frames, when the user clicks the shooting control subsequently, when the electronic device selects the corresponding RAW images from the RAW images output by the first camera and the RAW images output by the second camera, after the electronic device determines multiple frames of RAW images from the RAW images output by the first camera, the electronic device may not be able to completely find the RAW images with the same frame numbers as these multiple frames of RAW images in the RAW images output by the other camera.

[0020] Therefore, in this embodiment, the electronic device can halve the length of the second buffer queue to ensure that for any frame of RAW image in the second buffer queue, there is a RAW image with the same frame number in the first buffer queue. Then, the electronic device can first determine the frame numbers of the required multiple frames of RAW images from the second buffer queue, and then find another multiple frames of RAW images corresponding to the frame numbers of these multiple frames of RAW images from the first buffer queue, avoiding the situation where the camera needs to output frames again because the corresponding RAW images cannot be found.

[0021] Combined with the first aspect, in a possible embodiment, before saving the first photo, the method further includes: determining at least one frame of second RAW image from the RAW images output by the second camera; determining at least one frame of first RAW image from the RAW images output by the first camera based on the frame numbers of the at least two frames of RAW images, and the number of the at least one frame of first RAW image is the same as that of the at least one frame of second RAW image; fusing the at least one frame of first RAW image based on a multi-frame fusion algorithm to obtain a first image; processing the first key frame and the second key frame based on a binocular defocusing algorithm to obtain the depth of field information of the first key frame, where the first key frame is a frame of RAW image with better image quality among the at least one frame of first RAW images, and the second key frame is a RAW image with the same frame number as the first key frame among the at least one frame of second RAW images; performing defocusing processing on the background of the first image based on the depth of field information to obtain the first photo.

[0022] In this embodiment, since for any frame of RAW image in the second buffer queue, there exists a RAW image with the same frame number in the first buffer queue, the electronic device can first determine the at least one frame of second RAW images from the RAW images output by the second camera, and then it can also determine the same number of RAW images, i.e., the at least one frame of first RAW images, from the RAW images output by the first camera based on the frame numbers of the at least two frames of RAW images. To ensure the image quality of the final photo, the electronic device can determine, based on an image quality evaluation algorithm, a frame with better image quality among the at least one frame of first RAW images as the first key frame, and determine the RAW image with the same generation time as the first key frame among the at least one frame of second RAW images as the second key frame. Taking the first key frame as a reference, the information of other images among the at least one frame of first RAW images is superimposed on the first key frame to ensure the clarity of the subsequent obtained photo. After that, the electronic device can obtain the captured photo through a binocular blurring algorithm, that is, calculate the depth of field information of each pixel point in the first key frame through the first key frame and the second key frame, and accurately determine the foreground and background of the first key frame based on the depth of field information, and then blur the background image, which can accurately blur the background image while improving the clarity of the foreground image, avoiding the defects of missed background blurring or incorrect foreground blurring.

[0023] Combined with the first aspect, in a possible embodiment, the first camera may be a wide-angle camera, and the second camera may be an ultra-wide-angle camera; or, the first camera is a telephoto camera, and the second camera may be a wide-angle camera.

[0024] In a second aspect, the present application provides a photographing device, which includes a response unit, a first camera, and a second camera. The response unit is configured to start a portrait photographing mode in response to a first user operation, and the portrait photographing mode is a mode of jointly photographing using the first camera and the second camera; the first camera is configured to generate RAW images at a first frame rate, and the second camera is configured to generate RAW images at a second frame rate, and the first frame rate is greater than the second frame rate.

[0025] Specifically, the first camera and the second camera may be included in the same electronic device, and the first camera and the second camera can generate RAW images at the first frame rate and the second frame rate respectively under the control of the electronic device. Or, the first camera and the second camera include program codes or logic circuits, and the first camera and the second camera can control the frame rate of generating RAW images by themselves based on the program codes or logic circuits they contain.

[0026] In a third aspect, an embodiment of the present application provides an electronic device, where the electronic device includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0027] In a fourth aspect, a chip system is provided. The chip system is applied to an electronic device. The chip system includes one or more processors, and the processors are used to call computer instructions to cause the electronic device to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0028] In a fifth aspect, a computer program product containing instructions, when the computer program product runs on an electronic device, causes the electronic device to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0029] In a sixth aspect, a computer-readable storage medium is provided, including instructions, when the instructions run on an electronic device, causing the electronic device to execute the method in the first aspect or any possible implementation manner of the first aspect. Description of the Drawings

[0030] Figure 1 It is a schematic diagram of the architecture of a camera system provided by an embodiment of the present application;

[0031] Figure 2 It is a schematic diagram of the architecture of a camera system provided by an embodiment of the present application;

[0032] Figure 3 It is a flowchart of a shooting method provided by an embodiment of the present application;

[0033] Figures 4 - 6 It is some user interface diagrams provided by an embodiment of the present application;

[0034] Figure 7 It is a schematic diagram of an image output method of an electronic device provided by an embodiment of the present application;

[0035] Figure 8 It is a schematic diagram of a RAW image cache queue provided by an embodiment of the present application;

[0036] Figure 9 It is a schematic diagram of a scenario where a main and auxiliary two-way camera generates and outputs RAW images provided by an embodiment of the present application;

[0037] Figure 10 It is an Android system architecture diagram provided by an embodiment of the present application;

[0038] Figure 11 This is a structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0039] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular forms "a", "an", "the", "above-mentioned", "said", and "this" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more of the listed items.

[0040] For ease of understanding, the relevant terms involved in the embodiments of the present application will be introduced first below.

[0041] (1) Multi-camera device

[0042] A multi-camera device refers to an electronic device having two or more cameras. For example, the electronic device provided by the present application may include multiple cameras, which may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, etc., and may also include other types of cameras. These cameras may be arranged on the front or back of the electronic device, that is, as the front camera or the rear camera of the electronic device.

[0043] Compared with a single-camera device (i.e., an electronic device with only one camera), a multi-camera device has a wider shooting range and more viewing angle options, can capture a more comprehensive scene and multi-dimensional images, and the field of view is larger than that of a traditional single-camera device. A multi-camera device can synthesize the fields of view of multiple lenses and obtain a wider picture range. Specifically, a multi-camera device can use multiple camera modules to image separately, and through specific algorithm processing, fuse the images obtained by separately shooting by multiple modules into one image to achieve the purpose of specific imaging requirements. Secondly, a multi-camera device can achieve depth perception, that is, combined with depth measurement technology, allowing users to better perceive the depth of the scene while recording images. For example, when shooting a portrait, a multi-camera device can achieve background blurring according to distance data or make the portrait more realistic while blurring.

[0044] In this application, the image data output by each camera in a multi-camera device can be referred to as a data stream. When a user uses a multi-camera device to frame a shooting target, the data stream output by one camera on the multi-camera device can be used for display (i.e., displayed on the screen of an electronic device for the user to preview), and the data streams output by other cameras can be used for functions such as auxiliary information detection, registration, and ranging to correct or beautify the images in the data stream for display, thereby improving the quality of the displayed images. Among them, the camera whose output data stream is used for display can be called the "main camera", and the camera whose output data stream is used for auxiliary information detection, registration, and ranging can be called the "auxiliary camera".

[0045] However, the simultaneous output of data streams by multiple cameras will undoubtedly increase the power consumption of the electronic device and occupy the computing resources and memory resources of the electronic device to a greater extent.

[0046] (2) Background Blur

[0047] Background blur, also known as depth-of-field effect, is a camera technique that makes the depth of field shallower, causing the focus of the lens to be on the main subject of the picture and the background to appear blurred. It is most commonly used in the portrait mode provided by camera software. When taking a picture of an object in portrait mode, the electronic device can automatically blur the background to highlight the main subject.

[0048] Currently, the commonly used background blur techniques for electronic devices include monocular blur and binocular blur.

[0049] Monocular blur refers to a technique that achieves the blur effect through only one camera. For the monocular blur technique, the electronic device can control the degree and range of blur by adjusting the focal length and aperture size of the camera. It can be understood that when the focal length of the camera is far, the aperture is small, the depth of field is large, and the entire picture will be relatively clear; while when the camera is close, the aperture is large, the depth of field is small, only the objects near the focus will be clear, and the other parts will appear blurred. In this way, the electronic device can highlight the main subject in the image and blur the background by adjusting the focal length and aperture of the camera to achieve the blur effect.

[0050] Binocular defocusing requires a technology that achieves the defocusing effect through two cameras, so it needs to be applied to multi-camera devices. In addition, due to the limitation of the triangulation principle, it is necessary to calibrate the two camera modules so that their imaging planes are on the same plane and the pixels are aligned. That is to say, the two cameras used for shooting need to be on the same side of the electronic device, that is, both of these two cameras need to be the front cameras of the electronic device or both need to be the rear cameras of the electronic device. In a dual-camera system, one camera is responsible for shooting the subject, and the image data stream output by it can be used for display; while the other camera is responsible for measuring the depth-of-field information of each point, that is, the electronic device can determine the distance from each object in the scene to the electronic device (camera) through the image captured by this camera, and separate the foreground and background according to this information, and then perform defocusing processing on the background through the defocusing algorithm. Compared with monocular defocusing, binocular defocusing can defocus the subject more precisely and delicately, thus simulating the visual effect of the human eye to a greater extent and making the defocusing effect more natural.

[0051] (3) RAW images and YUV images

[0052] A RAW image is the raw data obtained by converting the light source signal captured by an image sensor into a digital signal, and it is the data format of the image output by the image sensor. The sensor can be called a photosensitive element, which is a device that converts an optical image into an electrical signal, such as a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In the field of image technology, the RAW format is an unprocessed and uncompressed format, that is, the raw image coding data (digital negative). Common suffixes of the RAW format include.ARW,.SRF,.SR2,.crw,.cr2,.cr3, etc.

[0053] A YUV image is a digital image format that separates the image brightness (Y) and colors (U, V), and is commonly used in the fields of video coding and digital image processing. In fact, YUV is a color coding method that separates the luminance information (Y) from the color information (UV), and a complete image can still be displayed without UV information, but this image is a black-and-white image.

[0054] It should be understood that the RAW output by the sensor Figure 1Generally, it will not be directly displayed on the display screen of the electronic device because the human eye generally cannot directly obtain scene information from the RAW image. This is because CMOS sensors are generally single-channel image sensors, and each pixel only records one channel signal. Therefore, the RAW image output by the sensor needs to be subjected to a series of operations including white balance correction, color space conversion, and tone mapping by the electronic device before the final YUV image presented on the display screen of the electronic device can be obtained.

[0055] (4) Zero Shutter Lag (ZSL)

[0056] In daily life, when using a mobile phone to take pictures, there are often some delayed experiences, that is, after the user clicks the shooting control on the mobile phone. Usually, the user needs to wait for a while before the electronic device outputs the captured image to the user. This is because in the non-ZSL mode, the electronic device starts previewing and outputs preview frames; when the user presses the shutter, the preview stream will be stopped, and at the same time, the electronic device needs to perform a series of actions, including af focusing, preparing for shooting, exposure, and callback of the shooting stream data. At this time, the obtained image is a RAW image. Then the electronic device will encode the RAW image to obtain a JPEG image and save it to the set storage area. At this time, the shooting is completed and the preview starts again. That is to say, in the non-ZSL mode, the preview is stopped during shooting, and the RAW image used in the preview stream is not stored in the electronic device. Only after the user presses the shutter, the electronic device will store several frames of RAW images and process them to obtain an image that can be browsed by the user (such as an image in JPEG format). For example, when the user shoots a target that is jumping, the user wants to shoot the moment when the target jumps and presses the shutter at the moment when the target jumps. However, in the non-ZSL mode, the electronic device can only use the RAW image captured by the camera after the user presses the shutter to process and obtain the final photo. After the user presses the shutter, the above target may have completed the jump and returned to the ground. Therefore, the RAW image captured by the camera after the user presses the shutter is not the image captured at the moment when the target jumps. It is very likely that the final photo shows not the picture of the target jumping but the picture of the target having landed. That is to say, in the non-ZSL shooting mode, there is a certain delay between the picture captured by the user and the picture that the user actually hopes to capture.

[0057] ZSL is a shooting mode developed to eliminate such delays and provide an experience of "instant viewing after shooting". In the ZSL mode, when the electronic device starts previewing, the raw images generated by the sensor are stored in the buffer. When the user presses the shutter, the device's system calculates the actual shooting time, locates the corresponding frame of raw image stored in the buffer, and then encodes the raw image to obtain a JPEG image and saves it to the set storage area. That is to say, in the ZSL mode, when the user clicks the shutter to take a photo, the preview can continue without stopping, and the raw image corresponding to the most recently used image in the preview stream is stored in the buffer. After the user presses the shutter, the system can calculate the delay time (i.e., the time difference between the time when the user actually wants to take a photo and the time when the user actually presses the shutter, which is the user's reaction time), then recognize a certain image frame as the frame for real-time shooting, and process it to obtain an image for the user to view (such as a JPEG-format image).

[0058] (5) Camera Structure

[0059] In this application, each camera of the electronic device can include an image sensor (sensor) and an image signal processor (ISP). Among them, the sensor is the photosensitive element described above, which can convert photons into electrical signals, and through an amplification circuit and an analog-to-digital conversion circuit, convert them into digital signals, that is, raw images. The ISP can include an image front end (IFE). During the operation of the camera, the raw images output by the sensor will first be transmitted to the IFE. The IFE can perform color correction, downsampling, demosaicing, etc. on the raw images to obtain YUV images; the IFE will further output the YUV images to the image processing engine (IPE), and the IPE performs image processing tasks such as hardware noise reduction, cropping, noise reduction, color processing, and detail enhancement on the YUV images to obtain the final YUV images for display (this image is for the user to preview, not the photo obtained after the user clicks the shooting control). If the current shooting mode of the electronic device is the ZSL mode, the IFE will further store the received raw images in the buffer. When the user clicks the shooting control, the electronic device can calculate the actual shooting time, locate the corresponding frame of raw image stored in the buffer, and then encode the raw image to obtain a JPEG image and save it to the set storage area. This JPEG image is the photo taken by shooting.

[0060] It should be understood that in addition to the above components, each camera in the present application may also include other components, such as lenses, focus motors, filters, etc., which are not limited in the present application.

[0061] (6) Frame rate

[0062] The frame rate is the frequency (rate) at which bitmap images, measured in frames, appear continuously on a display. In the embodiments of the present application, the frame rate may represent the number of images generated by a camera (or camera component) per second. For example, it may be used to represent the number of RAW images generated by a sensor in a camera per second. Specifically, assuming that the first sensor in the first camera of a multi-camera device generates one RAW image every 0.02 seconds, and the second sensor in the second camera generates one RAW image every 0.04 seconds, it can be considered that the frame rate of the first camera (first sensor) for generating RAW images is twice that of the second camera (second sensor) for generating RAW images.

[0063] (7) Application layer, application framework layer, hardware abstraction layer

[0064] The application layer (Application), application framework layer (Android Framework), and hardware abstraction layer (Hardware Abstraction Layer) are four layers in the Android system architecture. Among them:

[0065] The application layer includes system-built-in applications and non-system-level applications, which are usually developed based on the Java language and are mainly responsible for direct interaction with users. For example, when a user operates on an electronic device running on the Android system and opens a photo-taking application (such as a camera), the application layer can respond to the user operation, send the requirements to the application framework layer through the Camera Api v2 standard interface, and wait for the application framework layer to return the processing results, including image data and overall camera system status parameters. Then, the results are fed back to the user in a certain way. For example, through the SurfaceFlinger module, the position arrangement and content drawing of each layer in the image returned by the application framework layer are performed, so that the image is finally displayed on the screen of the electronic device.

[0066] The application framework layer is the foundation of Android application development. This layer is written in Java code and provides developers with the APIs needed to develop applications. Many core applications also implement their core functions through this layer. This layer simplifies component reuse. Developers can directly use the components provided by it for rapid application development, or achieve personalized expansion through inheritance. Specifically, the application framework layer can include multiple parts such as the activity manager, window manager, content provider, view system, package manager, phone manager, resource manager, location manager, notification manager, etc.

[0067] The hardware abstraction layer encapsulates the underlying hardware drivers and provides a general interface for the application framework layer to call the drivers. As long as the manufacturer implements the corresponding interfaces according to the specifications and stores them in a specific directory in the form of a shared library, then the upper layer only needs to load this shared library and find the pointer to the device corresponding to the corresponding module, and can operate the underlying hardware after obtaining the pointer to the entire device. For example, the aforementioned application framework layer can send the frame request instruction received from the application layer to the hardware abstraction layer, so that the sensor in the hardware abstraction layer generates a frame of RAW image according to this instruction, and passes the RAW image to the IFE. After the IFE converts it into a YUV image, it further passes the YUV image to the IPE for processing. Finally, the obtained image can be passed back to the application layer through the application framework layer for display.

[0068] In order to pursue better image quality and create various image effects for images, most electronic devices nowadays are usually equipped with multiple cameras. For example, the rear cameras of a mobile phone can include multiple cameras such as a wide-angle camera, an ultra-wide-angle camera, and a telephoto camera. In many shooting scenarios, a multi-camera device will use two cameras to shoot simultaneously. For example, in portrait mode, a multi-camera device can shoot the overall picture through one camera, and the data stream output by this camera will be used for display; at the same time, the multi-camera device can also obtain depth information through another camera, blur the background outside the main body in the image, and thus highlight the main body in the image.

[0069] Figure 1 Shows the working mode of the camera system 10 in a multi-camera electronic device in the ZSL mode and portrait mode.

[0070] As Figure 1As shown, the camera system 10 may include a camera 101, a camera 102, and an image processing engine 103. The camera 101 may include an image sensor 1011 and an image front end 1012, and the camera 102 may include an image sensor 1021 and an image front end 1022. In the embodiments of the present application, the camera 101 may be referred to as the main road camera, and the camera 102 may be referred to as the auxiliary road camera. In some embodiments, the image processing engine 103 may work in cooperation with the camera 101 and the camera 102 respectively under the control of software code. The processing logics when the image processing engine 103 works in cooperation with the camera 101 and when it works in cooperation with the camera 102 may be different, and the two can be carried out simultaneously without affecting each other.

[0071] After the user opens a shooting application (such as a camera), the cameras 101 and 102 may output images simultaneously.

[0072] Specifically, the image sensor 1011 of the camera 101 may collect optical signals at a certain acquisition rate, convert the optical signals into RAW images, and send the generated RAW images to the image front end 1012. Since the electronic device needs to use the RAW images stored in the buffer to generate photos in the ZSL shooting mode, after the image front end 1012 receives the RAW images transmitted by the image sensor 1011, on the one hand, the image front end 1012 will perform processes such as color correction and demosaicing on the received RAW images, convert the RAW images into YUV images, and deliver the YUV images to the image processing engine 103 at a frame rate of fps1 (that is, the image front end 1012 sends fps1 frames of RAW images to the image processing engine 103 per second, the same below); on the other hand, the image front end 1012 will also output the RAW images to the buffer of the electronic device at the frame rate fps1, that is, Figure 1 into the main road RAW image buffer queue in. After the image processing engine 103 receives the YUV images sent by the image front end 1012, it will further perform processes such as noise reduction and cropping on the YUV images to obtain the final main road YUV images for display.

[0073] At the same time, the image sensor 1021 of the camera 102 may also collect optical signals at a certain acquisition rate, convert the collected optical signals into RAW images, and send the generated RAW images to the image front end 1022. Similarly, after the image front end 1022 receives the RAW images transmitted by the image sensor 1021, the image front end 1022 will process and convert the received RAW images into YUV images, and also send the YUV images to the image processing engine 103 at a frame rate of fps1; on the other hand, the image front end 1022 will output the RAW images to the buffer of the electronic device at the frame rate fps1, that is, Figure 1in the secondary road RAW image cache queue. After the image processing engine 103 receives the YUV image sent by the image front end 1022, it will further perform noise reduction, cropping and other processing on the YUV image to obtain a secondary road YUV image for assisting the display of the aforementioned main road YUV image.

[0074] Here, it should be noted that the number of RAW images generated by the image sensor 1011 per second is the same as that generated by the image sensor 1021 per second, and the number of RAW images sent by the image sensor 1011 to the image front end 1012 per second is also the same as the number of RAW images sent by the image sensor 1021 to the image front end 1022 per second; in addition, both the image front end 1012 and the image front end 1022 will send YUV images to the image processing engine 103 and the image processing engine 103 respectively at the same frame rate fps1, and output RAW images to the cache at the same frame rate fps1 respectively.

[0075] In the real-time preview stage, the electronic device can further process the main road YUV image output by the image processing engine 103 through a binocular blurring algorithm to obtain an image with a blurring effect finally used for display on the screen for the user to preview. Specifically, the electronic device can use the main road YUV image output by the image processing engine 103 as a reference and combine the YUV Figure 1 starting from it to calculate the depth of field information of each pixel point in the main road YUV image, and then determine the foreground and background in the main road YUV image, and perform background blurring processing on the main road YUV image. The YUV image obtained after the blurring processing can be sent to the screen of the electronic device for the user to preview.

[0076] In the shooting stage, that is, after the user clicks the shooting control (shutter), the electronic device can select several frames of historical cached main road RAW images from the main road RAW image cache queue according to the shooting time (i.e., the time when the user clicks the shooting control), and select the same number of RAW images with the same generation time as the main road RAW images from the secondary road RAW image cache queue based on the selected main road RAW images, and then perform multi-frame fusion after blurring processing to obtain the final photo stored in the electronic device.

[0077] Optionally, the electronic device may determine three RAW images P1, P2, and P3 from the above-mentioned main path RAW image cache queue, and determine corresponding three RAW images P1', P2', and P3' from the above-mentioned secondary path RAW image cache queue based on P1, P2, and P3. The cache numbers for storing P1, P2, and P3 are respectively the same as those for storing P1', P2', and P3' (that is, the image sensor 1011 and the image sensor 1021 respectively generate P1 and P1' by performing exposure processing on the shooting scene at the same moment, generate P2 and P2' by performing exposure processing on the shooting scene at another moment, and so on). The electronic device may calculate the depth of field information of each pixel point in P1 based on P1 and P1', determine the foreground and background of P1 based on the depth of field information, and then blur the background in P1 to obtain an image with a blurring effect; similarly, the electronic device may obtain another two images with a blurring effect based on P2 and P2', P3 and P3'. Then, the electronic device may use one of the images with a blurring effect as a key frame, superimpose the information in the other two images with a blurring effect onto the key frame through a multi-frame fusion algorithm, and finally store the obtained image as a photo in the electronic device.

[0078] Optionally, in some embodiments, after the user clicks the shooting control, both the image sensor 1011 and the image sensor 1021 can continue to output RAW images and continue to deliver them to the image front-end 1012 and the image front-end 1022 respectively. The image front-end 1012 and the image front-end 1022 can also continue to output the received RAW images and store them in the above-mentioned main path RAW image cache queue and the above-mentioned secondary path RAW image cache queue. Then, the electronic device can determine six frames of RAW images, namely P1, P2, P3, P4, P5, and P6, from the above-mentioned main path RAW image cache queue (where P1, P2, and P3 are the RAW images cached before the user clicks the shooting control, and P4, P5, and P6 are the RAW images cached after the user clicks the shooting control), and determine the corresponding six frames of RAW images, namely P1', P2', P3', P4', P5', and P6', from the above-mentioned secondary path RAW image cache queue based on P1, P2, P3, P4, P5, and P6 (where P1', P2', and P3' are the RAW images cached before the user clicks the shooting control, and P4', P5', and P6' are the RAW images cached after the user clicks the shooting control); then, the electronic device can determine the key frame from the six frames of RAW images P1, P2, P3, P4, P5, and P6 through image quality evaluation (here it is assumed that the P1 image has the best quality and is determined as the key frame), and use the remaining frames (i.e., P2 - P6) as reference frames. First, the information in all reference frames is superimposed on the key frame through a multi-frame fusion algorithm to obtain an image P7 fused from multiple frames. Then, the depth of field information of each pixel point in P1 is calculated through P1 and P1' (i.e., the reference frame and the corresponding secondary path RAW image of the reference frame), and the foreground and background of P7 are determined based on the depth of field information. Furthermore, the background in P7 is blurred to obtain an image P8 with a blurred effect, and this image P8 is stored in the electronic device as the captured photo. Optionally, after the user clicks the shooting control, the electronic device can use other methods to process the historical cached RAW images to obtain the above-mentioned photo, which can be specifically determined by the algorithm logic of the multi-frame fusion algorithm and the binocular blurring algorithm adopted by the electronic device, and this application does not limit this. However, to ensure the background blurring effect of the photo and that there is no excessive delay between the photo image and the user's shooting moment, the historical cached RAW images (i.e., the RAW images stored before the user clicks the shooting control) obtained by the electronic device must simultaneously include the historical cached RAW images of the camera 101 and the historical cached RAW images of the camera 102.

[0079] As can be seen from the above description, in the ZSL shooting mode, in order to achieve a more precise background blur effect for images in the portrait mode, the two cameras included in the multi-camera electronic device will use the binocular blur technology to blur the images during both preview and shooting. In addition, during preview, both cameras need to output YUV images at the same frame rate for the user to preview, and store the RAW images in the buffer queue at the same frame rate, so that the electronic device can select frames and output images with zero delay after the user clicks the shooting control. However, for a multi-camera electronic device, the continuous collaborative work of multiple cameras will increase the power consumption of the electronic device. Especially when using multiple cameras to shoot portraits, the chain involved in the portrait algorithm is too long, imposing a greater burden on the memory resources and computing power of the electronic device.

[0080] To address the above problems, the present application provides a shooting method. Under the condition of ensuring the frame rate of the main camera's image output, the method reduces the working intensity of the auxiliary camera by reducing the frame rate of the auxiliary camera's image output during the preview stage, thereby reducing the power consumption of the electronic device during shooting and the burden on the computing power and memory of the electronic device.

[0081] It should be noted in advance that although the blur effect presented by the monocular blur algorithm is not as precise as that presented by the binocular blur algorithm, the monocular blur algorithm requires lower computing power and power consumption for the device than the binocular blur algorithm. Moreover, what the user ultimately needs to obtain is the photo taken rather than the preview image displayed during preview. Therefore, in this method, reducing the frame rate of the auxiliary camera's image output during the preview stage can include two aspects: 1. During the preview stage, the auxiliary camera reduces the output frame rate of the RAW image, that is, reduces the rate at which the image sensor in the auxiliary camera generates the RAW image; correspondingly, the frame rate at which the image sensor transmits the RAW image to the image front end and the frame rate at which the image front end outputs and caches the RAW image in the buffer queue will also decrease. 2. During the preview stage, the auxiliary camera does not output the YUV image. That is, the image sensor of the auxiliary camera can still output the RAW image to the image front end, but the image front end will not output the converted YUV image to the image processing engine, and the image processing engine (auxiliary camera) will not output the YUV image either. The images displayed in the preview stream are all processed based on the monocular blur algorithm for the YUV image output by the main camera, that is, sacrificing the quality of the images in the preview stream to reduce the power consumption of the electronic device.

[0082] It should be understood that in the embodiments of the present application, the frame rate at which the image sensor in the auxiliary road camera generates the RAW image is only smaller than the frame rate at which the image sensor in the main road camera generates the RAW image. However, after the RAW image is generated by the image sensor in the auxiliary road camera, the generated RAW image will still be transmitted to the image front end of the auxiliary road camera, and the image front end of the auxiliary road camera will also output all the received RAW images to the buffer queue. In this way, in the ZSL mode, even if the electronic device processes the YUV image output by the main road camera based on the monocular defocusing algorithm during the preview stage to obtain the preview image, during the shooting stage, the electronic device can still obtain the images cached by the auxiliary road camera in the past from the buffer queue. Then, the electronic device can use the binocular defocusing algorithm to obtain a photo with a better defocusing effect.

[0083] Figure 2 The specific working mode of the camera system of the electronic device provided in the present application in the ZSL mode is shown. As Figure 2 shown, the camera system 20 may include a camera 201, a camera 202, and an image processing engine 203. The camera 201 may include an image sensor 2011 and an image front end 2012, and the camera 202 may include an image sensor 2021 and an image front end 2022. In some embodiments, the image processing engine 203 may work in cooperation with the camera 201 and the camera 202 respectively under the control of software code. The processing logics when the image processing engine 203 works in cooperation with the camera 201 and when the image processing engine 203 works in cooperation with the camera 202 may be different, and the two may be carried out simultaneously without affecting each other.

[0084] Among them, the camera 201 may be referred to as the main road camera, and the camera 202 may be referred to as the auxiliary road camera. Both the camera 201 and the camera 202 are front cameras, or both the camera 201 and the camera 202 are rear cameras. The types of the camera 202 and the camera 201 are not limited in the present application. Specifically, the camera 201 may be a wide-angle camera, and the camera 202 may be an ultra-wide-angle camera; or, the camera 201 may be a telephoto camera, and the camera 202 may be a wide-angle camera. After the user opens the shooting application (such as the camera), the camera 201 and the camera 202 may work in different ways.

[0085] Specifically, the image sensor 2011 of the camera 201 collects optical signals at a certain acquisition rate and converts the optical signals into RAW images at a frame rate of fps2. That is, the image sensor 2011 generates fps2 frames of images per second and continuously transmits these RAW images to the image front end 2012. After receiving the RAW images transmitted by the image sensor 2011, the image front end 2012 performs processing such as color correction and demosaicing on the received RAW images, converts the RAW images into YUV images, and also sends the YUV images to the image processing engine 203 at a frame rate of fps2. The image processing engine 203 performs processing such as noise reduction and cropping on the YUV images to obtain the main path YUV images for display. In addition, the image front end 2012 also outputs the RAW images to the cache of the electronic device at a frame rate of fps2, that is Figure 2 into the main path RAW image cache queue in

[0086] While the camera 201 is working, the camera 202 can also work together. Specifically, the image sensor 2021 in the camera 202 can also collect optical signals at a certain acquisition rate and convert the collected optical signals into RAW images at a frame rate of fps3. That is, the image sensor 2021 generates fps3 frames of images per second and continuously transmits these RAW images to the image front end 2022. After the image front end 2022 receives the RAW images transmitted by the image sensor 2021, the image front end 2022 can also perform processing on the RAW images to convert them into YUV images. However, the image front end 2022 may not send the converted YUV images to the image processing engine 203 (at this time, the YUV image transmission channel between the image front end 2022 and the image processing engine 203 is in a closed state); however, the image front end 2022 outputs the RAW images to the cache of the electronic device at a frame rate of fps3, that is Figure 2 into the auxiliary path RAW image cache queue in

[0087] Specifically, the electronic device can control the image sensor 2011 and the image sensor 2021 to generate RAW images at different frame intervals. Among them, the frame interval for the image sensor 2011 to generate RAW images is smaller than that of the image sensor 2021 to generate RAW images. For example, the image sensor 2011 can generate one frame of RAW image every 1 / 30 second under the control of the electronic device, while the image sensor 2021 can generate one frame of RAW image every 1 / 15 second under the control of the electronic device. Both the image sensor 2011 and the image sensor 2021 can respectively transmit each generated frame of RAW image to the image front end 2012 and the image front end 2022. It can be understood that since the frame rates of the images generated by the image sensors included in the two cameras are different, the number of RAW images sent by the image sensor 2011 to the image front end 2012 per second is also different from the number of RAW images sent by the image sensor 2021 to the image front end 2022 per second; similarly, the number of RAW images sent by the image front end 2012 and the image front end 2022 to the image processing engine 203 per second is also different, which is determined by the frame rates of the images generated by the image sensors included in the two cameras respectively.

[0088] Optionally, the value of the above fps3 can be 1 / 2 of the value of the above fps2. Or, the value of the above fps3 can also be 1 / 3 of the value of fps2, or other values smaller than the value of fps2, and the present application does not limit this. Therefore, it can be understood that Figure 2 the number N of RAW images in the main path RAW image cache queue in Figure 2 will be greater than the number n of RAW images in the secondary path RAW image cache queue. Taking the value of fps3 as 1 / 2 of the value of fps2 as an example, within the same duration, the above N is equal to 2n.

[0089] In the real-time preview stage, the electronic device can further process the main path YUV image output by the image processing engine 203 through a monocular defocusing algorithm to obtain the finally displayed image with a defocusing effect on the screen for the user to preview. Specifically, the electronic device can identify the main body in the main path YUV image through an AI algorithm, then defocus the background, and transmit the defocused YUV image to the screen of the electronic device for the user to preview.

[0090] In the shooting stage, the electronic device can select several frames of historically cached RAW images from the main path RAW image cache queue and the secondary path RAW image cache queue respectively according to the shooting time (i.e., the time when the user clicks the shooting control), and then fuse the multi-frame images with defocusing effects after defocusing processing to obtain the photo finally stored in the electronic device. For specific reference, please refer to the relevant descriptions of Figure 1 and the relevant descriptions of the subsequent embodiments, which will not be elaborated here.

[0091] Of course, in some embodiments, if a preview image with a better blurring effect needs to be displayed for the user during the real-time preview phase, the electronic device can also control the image front end 2022 in the auxiliary road camera (i.e., camera 202) to transmit a YUV image to the image processing engine 203 during the preview phase (at this time, the YUV image transmission channel between the image front end 2022 and the image processing engine 203 is in an open state), and only by adjusting the frame rate of the RAW image generated by the auxiliary road camera, make the frame rate of the RAW image generated by the auxiliary road camera less than the frame rate of the RAW image generated by the main road camera. In this way, the electronic device can provide a preview image with a better blurring effect for the user during the preview phase, and can also reduce the working intensity of the auxiliary road camera to a certain extent, thereby reducing the power consumption of the electronic device.

[0092] Combined with the foregoing description of the imaging system 20, the shooting method provided by the present application will be introduced next. Please refer to Figure 3 .

[0093] Figure 3 is a flowchart of a shooting method provided by an embodiment of the present application. Figure 3 Taking the example that the preview stream uses a monocular blurring algorithm, the photo-taking uses a binocular blurring algorithm, and the frame rate of the RAW image generated by the auxiliary road camera is half of the frame rate of the RAW image generated by the main road camera, the shooting method provided by the present application is illustrated. As Figure 3 shown, in the shooting method provided by the embodiment of the present application, the first acquisition module and the first IFE module can be included in the first camera, and the first camera can be the main road camera in this method; the second acquisition module and the second IFE module can be included in the second camera, and the second camera can be the auxiliary road camera in this method. Among them, the first camera can be the foregoing camera 201, and the second camera can be the foregoing camera 202. The present application does not limit the specific types of the first camera and the second camera. Specifically, the first camera can be a wide-angle camera, and the second camera can be an ultra-wide-angle camera; or, the first camera can be a telephoto camera, and the second camera can be a wide-angle camera.

[0094] When implementing this method, the camera application in the application layer can load the portrait mode in response to the user's operation of launching the camera application. After the portrait mode is loaded, the user can start the portrait mode by touching the portrait mode icon. Then, the hardware abstraction layer can identify the shooting scene and report it to the shooting control module in the application layer. The shooting control module can adjust the shooting parameters and shooting methods in the portrait mode and send them back to the first acquisition module and the second acquisition module in the hardware abstraction layer. Finally, the first acquisition module and the second acquisition module can acquire images according to the adjusted shooting parameters and shooting methods. The first IFE module, the second IFE module, and the IPE module can determine the image processing algorithm to be used according to the identified shooting scene, and use this image processing algorithm to process the acquired images. The image data stream obtained after processing can be encoded to obtain an image file. The preview display module can also obtain the image data stream obtained after processing and perform preview display.

[0095] In this method, the IPE module can work in cooperation with the first camera and the second camera respectively under the control of software code. The processing logic when the IPE module works in cooperation with the first camera and the processing logic when the IPE module works in cooperation with the second camera can be different, and the two can be carried out simultaneously without affecting each other.

[0096] It can be understood that in Figure 3 , the data (including instructions, information, and images) transmission between the application layer and the hardware abstraction layer can be passed through the application framework layer (Android Framework). For the sake of convenience of description, Figure 3 the process of information transmission by the application framework layer is omitted. The shooting method can include but is not limited to the following steps:

[0097] S101. The user starts the camera application.

[0098] In the embodiments of the present application, the user can start the camera application by operating the application icon of the camera application, such as a touch operation.

[0099] S102. The mode loading module loads the mode.

[0100] When the camera application is started, the mode loading module can query the mode from the hardware abstraction layer. In the embodiments of the present application, the hardware abstraction layer can provide the portrait mode for the camera application. That is, in the portrait mode, in the hardware abstraction layer, the first acquisition module and the second acquisition module, the first IFE module and the IPE module, and the second IFE module and the second IFE module can be started to execute their respective functions.

[0101] In the embodiments of the present application, the hardware abstraction layer can also provide other modes for the camera application, such as the normal mode, the night scene mode, and the video recording mode, etc. The embodiments of the present application do not limit this.

[0102] Specifically, the mode loading module can query the hardware abstraction layer for modes. In response to the query from the mode loading module, the hardware abstraction layer can feedback to the mode loading module the modes provided by the hardware abstraction layer for the camera application. For example, the provided modes include: portrait mode, normal mode, night scene mode, video recording mode, etc.

[0103] Among them, the loaded modes include the portrait mode. During the loading process, the mode loading module also initializes the corresponding modules of each mode in the application layer and the hardware abstraction layer. After initialization, the electronic device can display the icons corresponding to each mode. For specific reference, please refer to the subsequent description. After initialization, in response to the user's touch operation on the icon corresponding to the portrait mode, the shooting control module can notify the first acquisition module, the second acquisition module, the first IFE module, the second IFE module, and the IPE module in the hardware abstraction layer to start and execute their respective functions. After initialization, other modes are similar to the portrait mode and can start the corresponding modules in the hardware abstraction layer in response to the user's touch operation on the icon corresponding to the mode.

[0104] Next, in combination with Figure 4 and Figure 5 the user interface involved in the process of loading the portrait mode will be introduced.

[0105] Please refer to Figure 4 , Figure 4 which is a schematic diagram of a human-computer interaction interface provided by an embodiment of the present application. As shown in (A) of Figure 4 , the electronic device can display the user interface 11. The user interface 11 includes a calendar widget 111, a weather widget 112, application icons 113, a status bar 114, and a navigation bar 115. Among them, the application icons 113 can include a gallery icon and a camera icon 1131, etc., and can also include icons of other applications. The embodiment of the present application does not limit this. Any application icon can be used to respond to the user's operation, such as a touch operation, so that the electronic device starts the application corresponding to the icon.

[0106] The user can start the camera application by touching the camera icon 1131. As shown in (A) of Figure 4 , in response to the user's touch operation on the camera icon, the mode loading module executes step S102. After the mode loading module finishes loading the modes, the electronic device can display the icons corresponding to each mode.

[0107] Exemplarily, the loaded modes include night scene mode, portrait mode, photo shooting mode, video recording mode, etc. As shown in (B) of Figure 4 , the electronic device can display the camera application interface 21. As shown in Figure 4As shown in (B) thereof, the camera application interface 21 may further include an echo control 212 for captured images, a shooting control 213, a camera switching control 214, a viewfinder 215, a focus control 216A, a setting control 216B, and a filter control 216C. Among them:

[0108] The shooting control 213 is used to respond to a user operation, shoot and save the captured photo.

[0109] The echo control 212 for captured images is used for the user to view the captured pictures and videos.

[0110] The camera switching control 214 is used to switch the camera for collecting images between the front camera and the rear camera.

[0111] The viewfinder 215 is used to perform real-time preview display on the collected pictures.

[0112] The focus control 216A is used to focus the camera.

[0113] The setting control 216B is used to set various parameters when collecting images.

[0114] The filter control 216C is used to select the filter effect during shooting.

[0115] The camera application interface 21 may include icons 211 corresponding to the loaded modes. The icons 211 may include a night scene mode icon 211A, a portrait mode icon 211B, a photo shooting mode icon 211C, a video recording mode icon 211D, and a more icon 211E. The shooting control module may respond to a touch operation of the user on any one of the icons 211 and start the mode corresponding to the icon. In the embodiment of the present application, the electronic device may, in response to a user operation, open the camera application and then display the camera application interface 21 on the display screen. The user may operate on any one of the above mode icons, such as a touch operation to start the corresponding shooting mode, and then the electronic device starts the corresponding module in the hardware abstraction layer.

[0116] S103. The user switches to the portrait mode.

[0117] As Figure 4 shown in (B) thereof, the user may touch the portrait mode icon 211B on the camera application interface 21 to switch to the portrait mode. In some embodiments, the operation of the user touching the portrait mode icon 211B on the camera application interface 21 to switch to the portrait mode may be referred to as a "first user operation".

[0118] It should be noted in advance that, in some embodiments, the background blurring effect of the electronic device on the image in the portrait mode may be turned on or off through corresponding operations. Taking Figure 5 as an example, through such asFigure 4 After the touch operation shown in (B) in Figure 5 switches the shooting mode to the portrait mode, the electronic device may display the user interface 31 shown in (A) in Figure 5 . As shown in (A) in

[0119] , the user interface 31 is the shooting interface of the portrait mode in the camera application. The user interface 31 may include a viewfinder 311, a background blurring control 312, and a beauty control 313. Among them:

[0120] The viewfinder 311 is used to display a real-time preview of the captured picture. A face image 3111 and a tree image 3112 are displayed in the viewfinder 311. When the background is blurred, the electronic device may determine the face image 3111 as the foreground (subject) and the tree image 3112 as the background.

[0121] The background blurring control 312 is used to blur the background of the image to highlight the foreground (subject) in the image.

[0122] As Figure 5 shown in (A) in Figure 5 , when the electronic device just starts the portrait mode, the background blurring control may default to the off state. Since the electronic device may not blur the background of the image in the preview stream at this time, the electronic device may not output the preview image according to the processing logic of the subsequent steps S106 - step S125. After responding to the user's touch operation on the background blurring control 312, the electronic device will, according to the processing logic in the subsequent steps S106 - step S125, display a preview image with a blurred background for the user in the viewfinder, and display the user interface 41 shown in (B) in

[0123] . It can be seen from the picture displayed in the preview frame 411 of the user interface 41 that the face image 4111, as the foreground, has a relatively clear overall picture and outline, while the background image such as the tree image 4112 has been blurred, so the face image 4111 is more prominent in the preview frame 411.

[0124] Optionally, in some embodiments, when the electronic device just starts the portrait mode, the background blurring control may also default to the on state. That is to say, after switching the shooting mode to the portrait mode through the touch operation shown in (B) in Figure 5 , the electronic device may directly blur the background of the image in the preview stream according to the processing logic of the subsequent steps S106 - step S125, and the electronic device may directly display the user interface 41 shown in (B) in Figure 5 .

[0125] S104. The shooting control module is started.

[0126] S105. The preview display module is started.

[0127] Taking (B) in the above as an example, the electronic device can, in response to the user's touch operation on the portrait mode icon 211B, start both the shooting control module and the preview display module. Figure 4 After the shooting control module and the preview display module are started, the shooting control module can enable the modules related to the portrait mode in the hardware abstraction layer, such as the first acquisition module and the second acquisition module. Specifically, the shooting control module can enable the first acquisition module and the second acquisition module in the hardware abstraction layer by sending a preview frame request instruction to the started hardware abstraction layer, and the first acquisition module and the second acquisition module can then start the image acquisition work.

[0128] In a possible implementation manner, the shooting control module and the preview display module may have been started in step S102, that is, in response to the user starting the camera application, the shooting control module and the preview display module are started. The shooting control module can be used for shooting control in various modes. The preview display module can be used for preview display in various modes.

[0129] It should be noted that in this application, the shooting control module can continuously send instructions for controlling the first acquisition module and the second acquisition module to generate RAW images. However, each request instruction generated by the shooting control module will be sent to the first acquisition module, but only one instruction out of every two request instructions generated by the shooting control module will be sent to the second acquisition module. Each time a request instruction generated by the shooting control module is received, the acquisition module will generate one frame of RAW image according to the instruction. Therefore, in this application, the frame rate of the first acquisition module generating RAW images is twice that of the second acquisition module generating RAW images.

[0130]

[0131] ​Here, all the operations performed by the first camera and the second camera under a request instruction issued by the shooting control module are referred to as a working cycle. It can be understood that, in the embodiments of the present application, since the shooting control module issues only one instruction to the second acquisition module for every two generated request instructions, the specific working logic of the second acquisition module will also be cyclically switched according to whether it receives a request instruction from the shooting control module. To introduce the specific working logics of the first camera and the second camera in detail, the following will describe the specific operation steps of the first camera and the second camera in two consecutive working cycles. Among them, steps S106 - S115 are the first working cycle, steps S107 - S111 are completed by the first camera, and steps S112 - S115 are completed by the second camera; steps S118 - S123 are the second working cycle, steps S119 - S123 are completed by the first camera, and in the second working cycle, the second camera does not output frames.

[0132] S106. The shooting control module sends a request instruction for starting image acquisition to the first acquisition module and the second acquisition module of the hardware abstraction layer.

[0133] S107. The first acquisition module generates a first RAW image.

[0134] S108. The first acquisition module sends the first RAW image to the first IFE module.

[0135] S109. The first IFE module outputs the first RAW image.

[0136] S110. The first IFE module converts the first RAW image into a first YUV image.

[0137] S111. The first IFE module sends the first YUV image to the IPE module.

[0138] It can be understood that steps S107 - S111 can be executed by the aforementioned first camera (main path camera, including the aforementioned first acquisition module and the first IFE module) and the IPE module. Among them, the first RAW image is generated by the above-mentioned first acquisition module and sent to the IPE module.

[0139] Specifically, the first acquisition module can generate the above-mentioned first RAW image according to preset shooting parameters. The preset shooting parameters may include any one or more of the following: shutter speed, exposure time, aperture value, exposure value, ISO. The shooting control module can set a shooting parameter and a shooting method for each shooting mode. Exemplarily, the shooting parameters set by the shooting control module in the portrait mode can be the first shooting parameters, and the shooting parameters set by the shooting control module in the night scene mode can be the second shooting parameters. After the electronic device activates the portrait mode, the shooting control module can send the shooting parameters corresponding to the portrait mode together with the enabling start instruction to the first acquisition module, so that the first acquisition module generates a RAW image according to the shooting parameters corresponding to the portrait mode.

[0140] Specifically, the first acquisition module can acquire the optical signal in the shooting scene at the above-mentioned first acquisition rate. After converting the optical signal into the above-mentioned first RAW image, the first acquisition module can send the first RAW image to the first IFE module. Since the electronic device needs to use the RAW images stored in the buffer to generate photos in the ZSL shooting mode, and the first camera is also responsible for outputting a preview image for the user to preview. Therefore, after the first acquisition module transports the first RAW image to the first IFE module, after receiving the first RAW image, the first IFE module will process the first RAW image in two aspects: on the one hand, the first IFE module will perform color correction, demosaicing and other processing on the first RAW image, convert the first RAW image into the above-mentioned first YUV image, and transport the first YUV image to the IPE module, and the IPE module will perform noise reduction, cropping and other processing on the YUV image to obtain the final YUV image for display. On the other hand, the first IFE module will output the RAW image to the buffer of the electronic device.

[0141] S112. The second acquisition module generates a second RAW image.

[0142] S113. The second module sends the second RAW image to the second IFE module.

[0143] S114. The second IFE module outputs the second RAW image to the IPE module.

[0144] S115. The second IFE module converts the second RAW image into a second YUV image.

[0145] It can be understood that steps S112 - S115 can be executed by the aforementioned second camera (auxiliary road camera, including the aforementioned second acquisition module, second IFE module) and the IPE module. Among them, the second RAW image is generated by the second acquisition module included in the above image acquisition module and sent to the IPE module.

[0146] Specifically, the second acquisition module can also generate the second RAW image according to the above preset shooting parameters. The shooting control module can set a shooting parameter and a shooting method for each shooting mode. After the electronic device activates the portrait mode, the shooting control module can send the shooting parameters corresponding to the portrait mode to the second acquisition module along with the enabling start instruction, so that the second acquisition module generates a RAW image according to the shooting parameters corresponding to the portrait mode.

[0147] It should be noted that the first RAW image and the second RAW image can be RAW images generated by the first acquisition module and the second acquisition module at the same moment respectively.

[0148] Specifically, the second acquisition module can also collect optical signals at a certain acquisition rate, convert the collected optical signals into the second RAW image, and send the second RAW image to the second IFE module. Combining the foregoing description, it can be seen that in order to reduce power consumption, the electronic device may not adopt the binocular blurring algorithm during the real-time preview stage, but adopt the monocular blurring algorithm to perform background blurring on the preview image, and the blurring of the image background does not need to be completed by means of the YUV image output by the second camera. Therefore, after the second IFE module receives the second RAW image, after converting the second RAW image into the second YUV image, the second IFE module will not send the second YUV image to the IPE module. However, in order to ensure the blurring effect of the photos taken in the ZSL shooting mode, the electronic device needs to process the image using the binocular blurring algorithm to obtain the final photo for saving. Then, after the user clicks the shooting control later, the electronic device needs to simultaneously obtain the RAW images generated by the first camera and the second camera in the past. Therefore, after the second IFE module receives the second RAW image, after converting the second RAW image into the second YUV image, the second IFE module will still output the second RAW image to the cache for the electronic device to use when shooting and generating images later.

[0149] In addition, it should be noted that since steps S107 - S111 and steps S112 - S116 are executed by two different cameras respectively, the electronic device can execute steps S112 - S115 synchronously when executing S107 - S111.

[0150] S116. The IPE module sends the first preview image obtained by processing the first YUV image to the preview display module.

[0151] S117. The preview display module displays the first preview image.

[0152] The above first preview image is an image obtained by the electronic device processing the above first YUV image using the monocular blurring algorithm. Specifically, reference can be made to Figure 5The picture displayed in the preview box in (B).

[0153] S118. The shooting control module sends a request instruction for starting to capture an image to the first capture module of the hardware abstraction layer.

[0154] In order to control the frame rate of the second camera to generate a RAW image to be half of that of the first camera, different from the first working cycle, in the second working cycle, the request instruction generated by the shooting control module will only be sent to the first capture module and will not be sent to the second capture module.

[0155] S119. The first capture module generates a third RAW image.

[0156] S120. The first capture module sends the third RAW image to the first IFE module.

[0157] S121. The first IFE module outputs the third RAW image.

[0158] S122. The first IFE module converts the third RAW image into a third YUV image.

[0159] S123. The first IFE module sends the third YUV image to the IPE module.

[0160] Similarly, steps S120 - S121 can be executed by the aforementioned first camera. Among them, the third RAW image is generated by the first capture module included in the above image capture module and sent to the IPE module, and the third RAW image can be the latest RAW image generated by the first capture module after generating the above first RAW image.

[0161] Specifically, after generating the first RAW image, the above first capture module will continue to capture the optical signal in the scene and convert the optical signal into the above third RAW image. Therefore, the above first capture module will continue to send the above third RAW image to the first PIE module. Combining the foregoing description, it can be seen that after receiving the third RAW image, the first IFE module will process the third RAW image in two aspects, that is, the first IFE module will convert the third RAW image into the above third YUV image and send the third YUV image to the IPE module. In addition, the IPE module will output the RAW image to the cache of the electronic device.

[0162] As can be seen from step S107 to step S111 and step S119 to step S123, since each request instruction generated by the shooting control module is sent to the first acquisition module, the working logic of the first camera in all working cycles is the same, and a frame of RAW image is generated in each working cycle. In the subsequent shooting process, the first camera will generate and output images according to the operations shown in step S107 to step S111 (or step S119 to step S123).

[0163] Since in the second working cycle, the request instruction generated by the shooting control module is not sent to the second acquisition module, the second acquisition module will not generate and output RAW images in this application. In addition, as can be seen from step S106 and step S118, since the request instructions generated by the shooting control module are sent to the second acquisition module intermittently, the second camera will generate a frame of RAW image every other working cycle (that is, every two working cycles). In the subsequent shooting process, the second camera will periodically switch different working modes to control the frame rate of the RAW images generated by the second acquisition module to be half of that of the first acquisition module.

[0164] S124. The IPE module sends the second preview image obtained after processing the third YUV image to the preview display module.

[0165] S125. The preview display module displays the second preview image.

[0166] Similarly, the above second preview image is also the image obtained by the electronic device processing the above third YUV image using the monocular defocusing algorithm. For details, please refer to Figure 5 the picture shown in the preview box in (B).

[0167] After that, the first camera and the second camera in the electronic device can continue to capture images according to the image capture logic shown in the foregoing steps. Specifically, the shooting control module will send each generated request instruction to the first acquisition module, and the first acquisition module in the first camera will generate RAW images at the first frame rate. At the same time, the shooting control module sends a request instruction to the second acquisition module every other request instruction (that is, a request instruction is sent to the second acquisition module only every two generated request instructions), and the second acquisition module in the second camera generates RAW images at the second frame rate. At the same time, the first acquisition module will send the generated RAW images to the first IFE module mentioned above. The first IFE module converts each received frame of RAW image into a YUV image and sends it to the IPE module, and the first IFE module will output each received frame of RAW image to the cache of the electronic device; the IPE module performs cropping and monocular blurring processing on the received YUV image and then outputs it to the screen of the electronic device for the user to preview. Similarly, the second acquisition module will send the generated RAW images to the second IFE module mentioned above. After the second IFE module converts each received frame of RAW image into a YUV image, it will not send it to the IPE module (or the second IFE module may not convert the RAW image into a YUV image either). Correspondingly, when the IPE module cannot receive the YUV image sent by the IPE module, it will not output the YUV image either; however, the second IFE module will also output all the RAW images received from the second acquisition module to the cache of the electronic device.

[0168] For example, in the third working cycle after the second working cycle, the shooting control module will send the generated request instructions to the first acquisition module and the second acquisition module at the same time. Therefore, both the first acquisition module and the second acquisition module will generate one frame of RAW image under the control of this instruction; but in the fourth working cycle after the third working cycle, the shooting control module will send the generated request instructions only to the first acquisition module, and the first acquisition module will generate one frame of RAW image under the control of this instruction, but the second acquisition module will not generate RAW images. In this way, the electronic device can control the value of the second frame rate to be only half of the value of the first frame rate, so as to reduce the working intensity and power consumption of the second camera.

[0169] If the user clicks the shooting control later, such as Figure 5 the touch operation of the user on the shooting control 413 shown in (B) of, the electronic device can select several frames of RAW images from the RAW images output from the first IFE module to the cache, and at the same time select the same number of RAW images corresponding to the foregoing several frames of images from the RAW images output from the second IFE module to the cache. After performing image background blurring, multi-frame image fusion, etc. on the obtained RAW images, the last obtained frame of image is used as the captured photo (specifically, reference can be made to the subsequent description of Figures 6 - 8(The relevant description will not be elaborated here first) and stored in the electronic device.

[0170] Taking Figure 6 as an example, through the touch operation of the user on the shooting control 413 as shown in (B) of Figure 5 , the electronic device can display the user interface 51 as shown in (A) of Figure 6 . As shown in (A) of Figure 6 , the user interface 51 is the shooting interface of the portrait mode in the camera application. The user interface 51 may include a viewfinder 511, a background blurring control 512, and a captured image echo control 513. Among them:

[0171] The preview image will continue to be displayed for the user in real time in the viewfinder 511. A face image 5111 and a tree image 5112 are displayed in the viewfinder 511. When the background is blurred, the electronic device can determine the face image 5111 as the foreground (subject) and the tree image 5112 as the background. Since the background blurring control 512 is already in the on state at this time, the tree image 5112 has been blurred. However, since the preview image displayed by the electronic device blurs the image using a monocular blurring algorithm, the blurring effect of the preview image may be relatively rough, and there may be a situation where the background is not blurred or the subject is blurred. As shown in (A) of Figure 6 , the face image 5111 is the foreground, and its hair part 5111A also belongs to the foreground, but the hair part 5111A is still blurred.

[0172] The captured image echo control 513 can display the thumbnail of the above-mentioned photo obtained after the user clicks the shooting control.

[0173] In response to the touch operation of the user on the captured image echo control 513 as shown in (A) of Figure 6 , the electronic device can display the user interface 61 as shown in (B) of Figure 6 . The user interface 61 may be the application interface of the gallery application, which may include an image 611 for displaying the user's historical captured photos, that is, the electronic device processes a number of RAW images output by the above-mentioned first IFE module and the same number of RAW images output by the second IFE module to obtain the photo. In the image 611, the face image 6111 is determined as the foreground (subject), and the tree image 6112 is determined as the background. Combining the signing description, it can be seen that the image 611 blurs the image using a binocular blurring algorithm, so the blurring effect of the preview image is more delicate than that of the preview image. Among them, the outline of the face image 6111 is clear, and there is no part that is wrongly blurred. The tree image 6112 as the background image is also accurately blurred.

[0174] Figure 7 The specific image output mode when the electronic device implements the shooting method provided in this application in the ZSL mode is shown.

[0175] As Figure 7 (A) in shows the image output mode of the electronic device in the preview stage.

[0176] In the preview stage, the first camera outputs YUV images (such as the aforementioned first YUV image and third YUV image) for display at the first frame rate. These YUV images will ultimately be processed by the electronic device based on the monocular defocusing algorithm to blur the background, and then drawn on the screen of the electronic device for the user to preview. At the same time, the first camera also generates and outputs RAW images (such as the aforementioned first RAW image and third RAW image) for converting into the aforementioned YUV images to the cache of the electronic device at the first frame rate. That is to say, for each RAW image generated by the first camera, the first camera will output the RAW image to the cache of the electronic device; in addition, the first camera will also convert the RAW image into a YUV image and then send it to the screen of the electronic device (such as first sending it to the CPU or GPU, and then having the CPU or GPU draw it on the screen of the electronic device) for the user to preview. Taking Figure 7 images 701 and 702 in as an example (images 701 and 702 can be the first YUV image and the first RAW image in the aforementioned description respectively), image 702 can be a RAW image generated by the first acquisition module in the first camera, and image 701 can be a YUV image obtained after image 702 is processed by the first IFE module and IPE module in the first camera.

[0177] In the preview stage, since the YUV image input port of the IPE module (i.e., the data interface of the IPE module for receiving the YUV image transmitted by the second IFE module) is always in the closed state under the control of the electronic device, the second camera does not output YUV images for display throughout the process. And combined with the aforementioned description, it can be known that under the control of the electronic device, the frame rate of the first camera for generating RAW images is twice that of the second camera for generating RAW images. Therefore, for every 2 frames of RAW images generated by the first camera, only 1 frame of RAW image will be generated by the second camera. As Figure 7 (A) in shows that when the first camera generates image 702, the second camera also generates image 704 at the same time; however, when the first camera generates image 703, the second camera does not generate a RAW image; afterwards, when the first camera generates image 705, the second camera also generates image 706 at the same time, and so on. Similarly, the second camera will also output all the generated RAW images to the cache of the electronic device for use in the shooting stage.

[0178] As Figure 7(B) in it shows the way of image output of the electronic device during the shooting stage.

[0179] During the shooting stage, the first camera and the second camera can continue to output images in the same way as in the preview stage. However, in order to obtain the captured photo, the electronic device can select several frames from the RAW images output from the first camera to the buffer and the RAW images output from the second camera to the buffer for processing to obtain the above-mentioned photo. Specifically, the electronic device can determine image 707 and image 708 from the RAW images output from the first camera to the buffer, and determine image 709 and image 710 from the RAW images output from the second camera to the buffer based on the frame numbers of these two images (this number can reflect the generation time of the RAW image, and two RAW images with the same frame number are generated at the same time). Among them, the frame numbers of image 707 and image 708 are the same, and the frame numbers of image 709 and image 710 are the same. Then, the electronic device can perform quality evaluation on image 707 and image 708, select the image with higher image quality from them (here it is assumed that the image quality of image 708 is higher), and select the image with the same certificate number as image 708 from image 709 and image 710, that is, image 710. Then, the electronic device can use the binocular blurring algorithm to process image 708 and image 710 to obtain image 711 with a blurring effect. Correspondingly, the electronic device can use the binocular blurring algorithm to process image 707 and image 709 to obtain image 711 with a blurring effect, and store the finally obtained image 711 as the above-mentioned photo in the electronic device (such as in the gallery).

[0180] Of course, in some embodiments, the electronic device can also select a larger number of images (for example, three images are selected for each) from the RAW images output from the first camera to the buffer and the RAW images output from the second camera to the buffer respectively, and obtain the above-mentioned photo according to the above processing method. The present application does not limit this.

[0181] In addition, in some embodiments, the electronic device can also use the binocular blurring algorithm and the multi-frame fusion algorithm to obtain the above-mentioned first preview image and second preview image. For example, the electronic device can first determine the key frame and the reference frame in the RAW images output by the first camera, first fuse multiple frames of RAW images output by the first camera to obtain a fused image, and then calculate the depth of field information of the image based on the reference frame and the RAW image corresponding to the reference frame (that is, the RAW image output by the second camera and generated at the same time as the reference frame), and perform background blurring on the above-mentioned fused image based on this depth of field information, and store the blurred image as the above-mentioned photo in the electronic device (specifically, it can refer to the relevant description of Figure 1 above, which will not be repeated here).

[0182] Optionally, after the user clicks the shooting control, the electronic device may use other methods to process the historical cached RAW images to obtain the above-mentioned photo, which can be specifically determined by the algorithm logic of the multi-frame fusion algorithm and the binocular defocusing algorithm adopted by the electronic device, and this application does not limit this. However, to ensure the background defocusing effect of the photo and the absence of delay between the photo image and the user's shooting moment, the historical cached RAW images (i.e., the RAW images stored before the user clicks the shooting control) obtained by the electronic device must simultaneously include the RAW images historically output by the first camera and the RAW images historically output by the second camera.

[0183] In addition, as can be seen from the foregoing description, the RAW images output by the first IFE module and the second IFE module will be saved in the cache (hereinafter, the RAW image output by the first IFE module and saved in the cache is referred to as the RAW image output by the main camera, and the RAW image output by the second IFE module and saved in the cache is referred to as the RAW image output by the auxiliary camera). In addition, in order to be able to determine a pair of RAW images with the same generation time from the RAW images output by the two cameras during the subsequent shooting frame selection process, when outputting the RAW images to the cache, the electronic device will also add frame numbers to the RAW images. Among them, the RAW images generated at the same moment (such as the first RAW image and the second RAW image mentioned above) have the same corresponding frame numbers. When selecting RAW images, the electronic device can determine the required RAW images and their corresponding frame numbers from the RAW images output by one of the cameras, and then select the RAW images with the same frame numbers from the RAW images output by the other camera based on the frame numbers corresponding to these RAW images.

[0184] However, due to storage space limitations and the reason that the second IFE stores RAW images at intervals, when the user clicks the shooting control subsequently, when the electronic device selects the corresponding RAW images from the RAW images output by the main camera and the RAW images output by the auxiliary camera, after the electronic device determines multiple frames of RAW images from the RAW images output by the main camera, the electronic device may not be able to completely find the RAW images with the same frame numbers as these multiple frames of RAW images in the RAW images output by the other camera.

[0185] For Figure 8 illustration, it is assumed here that the electronic device stores 50 frames of RAW images generated by both the first camera and the second camera respectively, and their frame numbers are all 1-50. The maximum number of RAW images that can be stored in the first cache queue for storing the RAW images output by the first camera is 10, and the maximum number of RAW images that can be stored in the second cache queue for storing the RAW images output by the second camera is also 10. When there are new RAW images to be cached into the queue, the RAW image cached earliest in the queue will be cleared. Then, as Figure 8As shown in (A) therein, the first buffer queue caches 10 RAW images that are the latest output of the first camera. The frame numbers corresponding to the 10 RAW images cached in the first buffer queue are 41 - 50. The second buffer queue also caches 10 RAW images that are the latest output of the second camera. However, since the frame rate of generating RAW images of the second camera is only half of that of the first camera, the frame rate of the second camera outputting RAW images is also only half of that of the first camera. Therefore, the frame numbers corresponding to the 10 RAW images cached in the second buffer queue are 32, 34, 36, 38, 40, 42, 44, 46, 48, 50 respectively. After the user clicks the shooting control, assuming the electronic device needs to select 3 frames of images from the first buffer queue and the second buffer queue respectively and obtain the above - mentioned photo according to the processing logic in the foregoing description, and the electronic device selects three RAW images with frame numbers 46, 47, and 48 from the first buffer queue, then the electronic device needs to select three RAW images with the same frame numbers from the second buffer queue based on the frame numbers of these three RAW images. However, as can be seen from Figure 8 therein, there are only RAW images with frame numbers 46 and 48 in the second buffer queue, but there is actually no RAW image with frame number 47. Therefore, the electronic device may finally be unable to output the above - mentioned photo normally due to this.

[0186] To address the above - mentioned defect, in a possible implementation, the electronic device can halve the length of the second buffer queue and only retain the latest 5 stored RAW images to ensure that for any RAW image in the second buffer queue, there is a RAW image with the same frame number in the first buffer queue. Then, the electronic device can first determine the frame numbers of the three RAW images needed from the second buffer queue, and then find the other three RAW images corresponding to the frame numbers of these three RAW images from the first buffer queue.

[0187] As Figure 8As shown in (B) of , after halving the length of the cached data in the second cache queue, the frame numbers corresponding to the 10 most recently cached RAW images in the first cache queue are still 41-50, and the frame numbers corresponding to the 5 most recently cached RAW images in the second cache queue are 42, 44, 46, 48, and 50 respectively. It can be understood that the set {42, 44, 46, 48, 50} is included in the set {41, 42, 43, 44, 45, 46, 47, 48, 49, 50}. Therefore, after the user clicks the shooting control, no matter what the frame numbers of the required RAW images determined by the electronic device from the second cache queue are, the electronic device can select the RAW images with the same frame numbers from the first cache queue based on these frame numbers. For example, assuming that the frame numbers of the three RAW images required by the electronic device determined from the second cache queue are 44, 46, and 48 respectively, the electronic device can select three RAW images with frame numbers 44, 46, and 48 from the first cache queue, and obtain the above-mentioned photo through these 6 RAW images.

[0188] It can be understood that the embodiments of the present application are described by taking image shooting in portrait mode as an example, but the embodiments of the present application are not limited to portrait mode, and can also be used in other shooting modes involving multi-camera streaming. The embodiments of the present application do not make any limitations in this regard. In addition, in some embodiments, the frame rate of the second camera generating RAW images can also be 1 / 3, 1 / 4 or other values of the frame rate of the first camera generating RAW images, as long as the frame rate of the second camera generating RAW images is smaller than the frame rate of the first camera generating RAW images. The present application does not make any limitations in this regard.

[0189] It should be understood that in order to ensure the quality of the photos taken by the user, in the same working cycle, the time when the second camera generates RAW images needs to be as close as possible to the time when the first camera generates RAW images. Especially when there are moving targets in the shooting scene, the depth-of-field information provided by the images selected from the two cache queues will be accurate only when the image output times are the same, and the electronic device can better identify the foreground and background of the image and more accurately blur the image.

[0190] However, in some embodiments, due to the limitations in performance of the first camera and the second camera, although the electronic device can maintain the frame rate of the RAW images generated by the second camera at 1 / 2 of the frame rate of the RAW images generated by the first camera, there may be a significant gap between the time when the second camera generates RAW images and the time when the first camera generates RAW images in the same working cycle. For example, when the user starts the camera application and enters the portrait mode, the first camera and the second camera do not start at the same time due to performance differences, so they do not start collecting optical signals and generating RAW images at the same time, resulting in a significant gap between the time when the second camera generates RAW images and the time when the first camera generates RAW images in all subsequent working cycles; or due to performance or other reasons, the first camera and / or the second camera cannot always maintain the frame interval between generated images at the same value, and the frame interval of a certain RAW image generated at a certain moment is significantly different from the frame intervals between other RAW images, which will also cause a gap between the time when the second camera generates RAW images and the time when the first camera generates RAW images in all subsequent working cycles.

[0191] To address the above problems, in some embodiments, the electronic device can periodically select two images with the same frame number from the main and auxiliary RAW image buffer queues respectively, and determine whether the first camera and the second camera generate these two images at the same time based on the timestamp information of these two images. Since the frame rates of the RAW images output by the first camera and the second camera are determined, the frame intervals between the RAW images output by the first camera and the second camera are relatively stable (when there is no obvious fluctuation). Therefore, if there is a significant time interval (such as greater than or equal to 1 ms) between the times when the first camera and the second camera generate these two images respectively, the electronic device can adaptively adjust the frame interval between the latest RAW image generated by the image sensor in the second camera and the previous RAW image generated, that is, according to the above time interval, adaptively advance or delay the generation of the latest RAW image that should have been generated at a certain moment, so that the image sensors in the second camera and the first camera can generate the latest RAW images at the same time respectively. For details, please refer to Figure 9 .

[0192] Figure 9 Figure (A) in shows a scenario where there is a gap between the time when the second camera generates RAW images and the time when the first camera generates RAW images. As Figure 9As shown in (A) therein, the image 901 and the image 902, the image 903 and the image 904, the image 905 and the image 906, and the image 907 and the image 908 are RAW images generated by the first camera and the second camera in four different working periods. Here, it is assumed that the image 901 and the image 902 are RAW images respectively generated by the first camera and the second camera in the first period. Due to the performance difference between the two cameras, there is a time interval of T0 duration between the time t91 when the first camera generates the image 901 and the time t90 when the second camera generates the image 902, that is, in the first period, the first camera generates the RAW image T0 duration later than the second camera generates the RAW image (T0 duration is greater than or equal to 1 ms). Then it can be understood that without any intervention, in the third working period (the working period for generating the image 903 and the image 904), the fifth working period (the working period for generating the image 905 and the image 906), and the seventh working period (the working period for generating the image 907 and the image 908), there is a relatively large time interval between the time when the first camera generates the RAW image and the time when the second camera generates the RAW image as much as possible. Here, it is assumed that they are T1 duration, T2 duration, and T3 duration respectively. It can be understood that when the frame intervals of the RAW images generated by the first camera and the second camera in the subsequent are both stable, the T0 duration, T1 duration, T2 duration, and T3 duration can be the same. When there is one or more abnormal fluctuations in the frame intervals of the RAW images generated by the first camera or the second camera in the subsequent, the T0 duration, T1 duration, T2 duration, and T3 duration can be different.

[0193] From Figure 9 As can be seen from (A) therein, the four groups of images, namely the image 901 and the image 902, the image 903 and the image 904, the image 905 and the image 906, and the image 907 and the image 908, are not RAW images generated at the same moment. And without intervention, in the subsequent working periods, there may always be a relatively large time interval between the times when the first camera and the second camera generate RAW images. Then in the process of subsequent image capture, the main and auxiliary RAW images selected by the electronic device are not the images obtained by the first camera and the second camera shooting the scene at the same moment, which is very likely to result in poor quality of the finally obtained photo images.

[0194] Figure 9 (B) therein shows a scenario where the electronic device adjusts the time for the second camera to generate the RAW image in the fifth working period (the working period for generating the image 905 and the image 906), so that the time when the second camera generates the RAW image in the subsequent is as close as possible to the time when the first camera generates the RAW image.

[0195] It should be noted in advance that since the electronic device needs to determine the time interval between the moments when the first camera and the second camera respectively generate RAW images based on the timestamp information of two frames of images output by the main and auxiliary cameras stored in history, and adjust the moment when the image sensor in the second camera generates the latest frame of RAW image based on this interval. Therefore, both frames of images output by the main and auxiliary cameras obtained by the electronic device need to carry corresponding timestamp information. However, whether it is the first camera or the second camera, the RAW image it generates will carry timestamp information only after it is output to the cache outside the image sensor (such as the main RAW image cache queue and the auxiliary RAW image cache queue mentioned above). Usually, before the latest frame of RAW image is generated, the image sensor generally has not had time to output the latest 2-3 frames of images generated in history to the cache outside the image sensor, so these images will not carry the corresponding timestamp information. Taking Figure 9 in (A) as an example, at time t94, the image sensor in the second camera failed to output image 904, and at time t95, the image sensor in the first camera also failed to output image 903. Therefore, both image 903 and image 904 do not carry timestamp information. That is to say, if the electronic device needs to adjust the moment when the second sensor generates image 906 to shorten the time interval between the moment when the second camera generates the RAW image and the moment when the first camera generates the RAW image, the electronic device can only obtain the RAW images generated by the first camera and the second camera at an earlier moment and output to the cache queue and carrying timestamp information, such as image 901 and image 902, and determine the approximate time interval between the moment when the second camera currently generates the RAW image and the moment when the first camera generates the RAW image based on the timestamp information of these two frames of images, and adjust the moment when the image sensor in the second camera generates the latest frame of RAW image based on this time interval.

[0196] That is to say, in the embodiment of the present application, the electronic device can obtain the first N1 frames of RAW images generated by the first camera and the first N2 frames of RAW images generated by the second camera to determine the approximate time interval between the moment when the second camera currently generates the RAW image and the moment when the first camera generates the RAW image. Specifically, the value of N1 can be 4, and the value of N2 can be 2.

[0197] Such as Figure 9As shown in (B) in , during the fifth working cycle, the electronic device can obtain the images 901 and 902 generated by the first camera and the second camera and output to the buffer queue, and carrying timestamp information, and determine, based on the timestamp information of these two frames of images, that the time when the second camera generates image 902 is the t90 moment, and the time when the first camera generates image 901 is the t91 moment. Then the electronic device can determine that the time when the second camera generates the RAW image in the same working cycle is earlier than the time when the first camera generates the RAW image by a duration of T0. Therefore, during the fifth working cycle, according to the frame interval specified for the second camera to generate the RAW image and the time (i.e., the t92 moment) when it generates the previous RAW image (i.e., image 904), the second camera should originally generate image 906 at the t94 moment after the t92 moment under the control of the electronic device. However, under the intervention of the electronic device, the second camera will actually generate image 906 at the t94' moment after the t94 moment and at a duration of T0 from the t94 moment, and the electronic device will not change the time when the first camera generates image 905. It can be understood that under this kind of intervention, during the fifth working cycle, the time when the second camera generates the RAW image is later than before by a duration of T0. Then the time interval between the time when the second camera generates the RAW image and the time when the first camera generates the RAW image is also shortened from the original T2 duration to the (T2 - T0) duration, and the image output times of the two cameras can be closer; if T2 = T0, there may even be no time interval between the time when the second camera generates the RAW image and the time when the first camera generates the RAW image. In this case, the image output times corresponding to the images respectively selected by the electronic device from the two buffer queues later are closer, and the depth of field information obtained by it based on the two images is more accurate, and the electronic device can virtualize the images more precisely.

[0198] Optionally, during subsequent image generation, to save the computing power of the electronic device and reduce the device power consumption, the electronic device may use every 5 frames generated by the first camera as a cycle, and periodically obtain the first N1 RAW images generated by the first camera and the first N2 RAW images generated by the second camera, and determine the approximate time interval between the time when the second camera currently generates a RAW image and the time when the first camera generates a RAW image, and adjust the time when the second camera generates the latest RAW image based on this time interval, so that the time when the second camera generates a RAW image in the subsequent period is as close as possible to the time when the first camera generates a RAW image. For example, after adjusting the time when the second camera generates image 906 based on the timestamp information of image 901 and image 902 in the fifth working cycle, in the subsequent five working cycles (including three cycles when the second camera does not need to generate RAW images), the electronic device may no longer intervene in the time when the camera generates RAW images. Until the eleventh working cycle arrives, the electronic device can obtain image 907 and image 908 generated by the first camera and the second camera and output to the buffer queue and carrying timestamp information, and determine that the time when the second camera currently generates image 908 is the t96' moment based on the timestamp information of these two images (due to the intervention in the time when the second camera generates a RAW image in the fifth cycle, the time when the second camera generates image 908 has been delayed by a duration of T0 from the original t96 moment to the t96' moment in the seventh cycle), and the time when the first camera generates image 903 is the t97 moment, then the electronic device can determine that the time when the second camera generates a RAW image in the same working cycle is earlier than the time when the first camera generates a RAW image by a duration of (T3 - T0) (assuming that the duration of (T3 - T0) is greater than 1 ms). Therefore, in the eleventh working cycle, under the intervention of the electronic device, the second camera will delay the time when it generates the latest RAW image by a duration of (T3 - T0), and the electronic device will not change the time when the first camera generates the latest RAW image, and so on.

[0199] Alternatively, during the subsequent image generation process, before the second camera generates each latest frame of RAW image, the electronic device can obtain the first N1 frames of RAW images generated by the first camera and the first N2 frames of RAW images generated by the second camera, and determine the approximate time interval between the time when the first camera generates a RAW image and the time when the second camera generates a RAW image at the latest. Based on this time interval, the electronic device can adjust the time when the second camera generates the latest frame of RAW image, so that the time when the second camera generates a RAW image is as close as possible to the time when the first camera generates a RAW image in the subsequent process. For example, in the seventh working cycle, before outputting image 908, the electronic device can obtain image 903 and image 904 generated by the first camera and the second camera and output to the buffer queue, and carry timestamp information. Based on the timestamp information of these two frames of images, the electronic device can determine that the time when the second camera generates image 904 is time t92, and the time when the first camera generates image 903 is time t93. Then the electronic device can determine that the time when the second camera generates a RAW image in the same working cycle is earlier than the time when the first camera generates a RAW image by a duration of T1. Therefore, in the seventh working cycle, originally the second camera should generate image 906 at time t96, but under the intervention of the electronic device, the second camera generates image 906 at time t94” ( Figure 9 not shown in) after time t94 and at a time T1 duration away from time t96, and the electronic device does not change the time when the first camera generates image 905, and so on.

[0200] Figure 10 From the perspective of the Android system architecture, the process of the electronic device executing the above shooting method is shown. As Figure 10 shown, the architecture involved in the electronic device during the execution of the foregoing shooting method may include the following three layers: the application layer, the application framework layer, and the hardware abstraction layer. Among them, the specific functions of each layer of architecture in the Android system can refer to the foregoing relevant descriptions and will not be elaborated here. During the process of the electronic device executing the foregoing shooting method, the application layer, the application framework layer, and the hardware abstraction layer can work together to complete the entire preview process. Specifically:

[0201] The application layer can respond to user operations, such as the operation of the user opening the shooting application, and continuously send request instructions to the application framework layer. Each frame request instruction corresponds to a frame of preview image displayed on the screen of the electronic device. Alternatively, the application layer can respond to user operations, such as the operation of the user clicking the shooting control, and continuously send shooting image output request instructions to the application framework layer. The shooting image output request instructions can be used to instruct the hardware abstraction layer to obtain a photo using the cached RAW images.

[0202] After that, the application framework layer further sends the request instructions or the captured image request instructions transmitted by the application layer to the hardware abstraction layer. The hardware modules in the hardware abstraction layer output corresponding RAW images and / or YUV images according to the instructions. For specific details, reference can be made to the foregoing descriptions of Figure 2 and Figure 3 , which will not be elaborated here.

[0203] After that, the hardware layer can send the generated preview image (at this time, the preview image can be some drawing instructions) back to the application layer through the application framework layer. The SurfaceFlinger draws the preview image on the screen of the electronic device according to the drawing instructions for the user to view.

[0204] Next, the electronic device provided by the embodiments of the present application will be introduced.

[0205] The electronic device can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), or a dedicated camera (such as a single-lens reflex camera, a compact camera), etc. The present application does not impose any restrictions on the specific type of the electronic device. Specifically, the electronic device 100 can be the electronic device described above.

[0206] Figure 11 The structure of the electronic device is exemplarily shown.

[0207] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, a button 190, a motor 191, a camera 193, a display screen 194, etc.

[0208] It can be understood that the structure schematically shown in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware.

[0209] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0210] The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0211] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0212] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0213] The I2C interface is a two-way synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 can be respectively coupled to a charger, a flash, a camera 193, etc. through different I2C bus interfaces.

[0214] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100.

[0215] The USB interface 130 is an interface that conforms to the USB standard specification, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used for data transmission between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. This interface can also be used to connect other electronic devices, such as AR devices, etc.

[0216] It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present invention is only for illustrative purposes and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0217] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input of the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0218] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives inputs from the battery 142 and / or the charging management module 140 to supply power to the processor 110, the internal memory 121, the display screen 194, the camera 193, etc. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0219] The modem processor can include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to speakers, receivers, etc.), or displays an image or video through the display screen 194. In some embodiments, the modem processor can be an independent device. In some other embodiments, the modem processor can be independent of the processor 110 and be disposed in the same device as the mobile communication module or other functional modules.

[0220] The electronic device 100 realizes the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0221] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0222] The electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0223] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera sensor. The optical signal is converted into an electrical signal, and the camera sensor transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise and brightness of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0224] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the sensor. The sensor can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The sensor converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats. In some embodiments, the electronic device 100 may include N cameras 193, where N is a positive integer greater than 1.

[0225] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0226] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0227] The NPU is a neural-network (NN) computing processor. By learning from the biological neural network structure, such as learning from the transmission mode between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0228] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card.

[0229] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, the image playback function, etc.). The data storage area can store the data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory provided in the processor.

[0230] The keys 190 include a power-on key, volume keys, etc. The keys 190 can be mechanical keys or touch keys. The electronic device 100 can receive key inputs and generate key signal inputs related to the user settings and function control of the electronic device 100.

[0231] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations for different applications (such as taking pictures, audio playing, etc.) can correspond to different vibration feedback effects. For touch operations on different regions of the display screen 194, the motor 191 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminder, receiving messages, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0232] In this application, the camera 193 includes at least two cameras, including a first camera and a second camera. Both of these cameras are either the front cameras of the electronic device or the rear cameras of the electronic device. Optionally, the first camera can be a wide-angle camera and the second camera can be an ultra-wide-angle camera; or, the first camera can be a telephoto camera and the second camera can be a wide-angle camera.

[0233] In the embodiment of this application, the processor 110 can control the frame output mode of the first camera and the second camera in the ZSL mode.

[0234] In the preview stage of the ZSL shooting mode, the first ISP (including the first IFE) in the first camera can convert the RAW image generated by the first sensor in the first camera into a YUV image and output it, and the second ISP (including the second IFE) in the second camera can not output the RAW image generated by the second sensor in the second camera. The processor 110 can, after processing the YUV image output by the first camera based on the monocular defocusing algorithm, combine with the GPU to draw the processed image onto the display screen 194 for display.

[0235] In addition, in the preview stage of the ZSL shooting mode, the first camera generates and outputs RAW images at a first frame rate, and the second camera generates and outputs RAW images at a second frame rate, where the first frame rate is greater than the second frame rate. Optionally, the value of the first frame rate is twice the value of the second frame rate, that is, for every two frames of RAW images generated by the first image sensor in the first camera, the second image sensor in the second camera only generates one frame of RAW image.

[0236] The RAW images output by the first camera and the second camera can be cached in the cache area of the electronic device. This cache area can be a storage area in the memory included in the processor 110, or a storage area in an external memory, or a storage area in the internal memory 121.

[0237] In the shooting stage in the ZSL shooting mode, in response to a user's touch operation on the shooting control, the processor 110 may respectively select the same number of frames of RAW images from the RAW image output by the first camera and the RAW image output by the second camera, obtaining M groups of RAW images with the same generation time (i.e., two RAW images with the same frame number). The processor 110 may use a binocular defocusing algorithm to process each group of RAW images to obtain M frames of images (these images have a defocused background effect), and use a multi-frame fusion algorithm to fuse the M frames of images to obtain the final photo to be stored.

[0238] Optionally, after the user clicks the shooting control, the electronic device may use other methods to process the above photo based on the RAW images cached in history, which may be specifically determined by the algorithm logic of the multi-frame fusion algorithm and the binocular defocusing algorithm adopted by the electronic device, and this application does not limit this. However, to ensure the background defocusing effect of the photo and no delay between the photo image and the user's shooting moment, the RAW images cached in history obtained by the electronic device (i.e., the RAW images stored before the user clicks the shooting control) must simultaneously include the RAW images historically output by the first camera and the RAW images historically output by the second camera.

[0239] An embodiment of this application also provides an electronic device, which includes: one or more processors and a memory; wherein, the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method shown in the foregoing embodiment.

[0240] As used in the foregoing embodiments, depending on the context, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if detecting (the stated condition or event)" may be interpreted to mean "if determining..." or "in response to determining..." or "when detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)".

[0241] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive), etc.

[0242] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware with a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the above method embodiments. The foregoing storage medium includes: various media that can store program codes such as ROM or random access memory RAM, magnetic disks, or optical discs.

Claims

1. A shooting method, characterized in that, The method includes: In response to a first user operation, starting a portrait shooting mode, where the portrait shooting mode is a mode of jointly shooting with a first camera and a second camera; Controlling the first camera to generate a RAW image at a first frame rate, and controlling the second camera to generate a RAW image at a second frame rate, where the first frame rate is greater than the second frame rate.

2. The method according to claim 1, wherein The first frame rate is twice the second frame rate. The controlling the first camera to generate a RAW image at a first frame rate and controlling the second camera to generate a RAW image at a second frame rate includes: At a first moment, controlling the first camera to generate a first RAW image and controlling the second camera to generate a second RAW image; At a second moment, controlling the first camera to generate a third RAW image.

3. The method according to claim 2, wherein Before the controlling the first camera to generate a first RAW image and controlling the second camera to generate a second RAW image at the first moment, the method further includes: Determining a first duration based on a fourth RAW image generated by the first camera and a fifth RAW image generated by the second camera, where the frame numbers of the fourth RAW image and the fifth RAW image are the same, and the first duration is the time interval between the moment when the first camera generates the fourth RAW image and the moment when the second camera generates the fifth RAW image; Determining the first moment based on the first duration and the second frame rate.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Controlling the first camera to output a YUV image at the first frame rate, Processing the first YUV image output by the first camera based on a monocular defocusing algorithm to obtain a first preview image; Displaying the first preview image in a preview area of a display screen.

5. The method according to any one of claims 1 to 4, characterized in that The method is applied to a zero-second delay shooting mode. The method further includes: In response to a user operation on a shooting control, saving a first photo, where the first photo is obtained by processing at least one frame of the first RAW image generated and cached by the first camera and at least one frame of the second RAW image generated and cached by the second camera based on a binocular defocusing algorithm.

6. The method according to any one of claims 1 to 5, characterized in that The first frame rate is n times the second frame rate. The RAW images generated by the first camera are output to a first buffer queue, and the RAW images generated by the second camera are output to a second buffer queue. When both the first buffer queue and the second buffer queue are full, the number of RAW images stored in the second buffer queue is n times the number of RAW images in the first buffer queue, where n is a positive number.

7. The method according to claim 5 or 6, characterized in that, Before saving the first photo, the method further includes: Determining at least one frame of second RAW image from the RAW images output by the second camera; Determining at least one frame of first RAW image from the RAW images output by the first camera based on the frame numbers of the at least two frames of RAW images, where the number of the at least one frame of first RAW image is the same as the number of the at least one frame of second RAW image; Fusing the at least one frame of first RAW image based on a multi-frame fusion algorithm to obtain a first image; Process the first key frame and the second key frame based on the binocular defocusing algorithm to obtain the depth-of-field information of the first key frame. The first key frame is a RAW image with better image quality among the at least one frame of first RAW images, and the second key frame is a RAW image with the same frame number as the first key frame among the at least one frame of second RAW images; Blur the background of the first image based on the depth-of-field information to obtain the first photo.

8. The method according to any one of claims 1 to 7, wherein the first camera may be a wide-angle camera, and the second camera may be an ultra-wide-angle camera; alternatively, the first camera is a telephoto camera, and the second camera may be a wide-angle camera.

9. An electronic device, characterized in that, The electronic device includes: one or more processors, a memory, and a display screen; The memory is coupled to the one or more processors. The memory is used to store computer program code, and the computer program code includes computer instructions. The one or more processors call the computer instructions to cause the electronic device to execute the method according to any one of claims 1-8.

10. A chip system, characterized in that, The chip system is applied to an electronic device. The chip system includes one or more processors, and the processors are used to call computer instructions to cause the electronic device to execute the method according to any one of claims 1-8.

11. A computer-readable storage medium, comprising instructions, characterized in that, When the instructions run on the electronic device, the electronic device is caused to execute the method according to any one of claims 1-8.

Images