Shooting method and electronic equipment

Abstract

The invention provides a shooting method and electronic equipment, and the shooting method reduces the working intensity of an auxiliary road camera through reducing the picture output frame rate of the auxiliary road camera in a preview stage under the condition of maintaining the picture output frame rate of a main road camera, reduces the power consumption of the electronic equipment during shooting under the premise of maintaining the quality of a shot picture, and improves the shooting efficiency of the electronic equipment. And the computing power burden on the electronic equipment is reduced.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a shooting method and electronic equipment. Background Art

[0002] Nowadays, image capture and beautification have become an indispensable part of our daily life and entertainment. Nowadays, most electronic devices use two cameras to take pictures to create various image effects. For example, one of the cameras can be responsible for capturing the entire picture. The other camera can be used to obtain depth of field information, blur the background outside the subject in the image, and highlight the subject in the image. However, working with multiple cameras will increase the power consumption of electronic devices, and will also maximize the memory resources and computing power of electronic devices.

[0003] Therefore, how to reduce the power consumption of multi-camera equipment during the shooting process is an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present application is to provide a shooting method and an electronic device. The shooting method reduces the working intensity of the auxiliary camera by reducing the frame rate of the auxiliary camera in the preview stage while maintaining the frame rate of the main camera. The power consumption of the electronic device during shooting and the burden on the computing power of the electronic device are reduced while maintaining the quality of the taken photos.

[0005] The above-mentioned and other objects are achieved by the features of the independent claims. Further implementations are reflected in the dependent claims, the description and the drawings.

[0006] In a first aspect, the present application provides a shooting method, comprising: controlling a first camera to output a RAW image at a first frame rate; controlling a second camera to output a RAW image at a second frame rate; the first frame rate is greater than the second frame rate.

[0007] In this application, in order for the electronic device to use some binocular algorithms (such as binocular blur algorithm) to create specific image effects for the image when the user takes the image later, the electronic device needs to be able to simultaneously obtain the RAW images stored in the history of the two cameras after the user clicks to shoot. Therefore, in this application, the first camera and the second camera can output RAW images at the same time during the preview stage, and the output RAW images will be output to the cache.

[0008] In this method, the RAW image output by the first camera can be processed and displayed for user preview, and the RAW image output by the second camera can be used as auxiliary information to add specific image effects to the displayed image. In order to reduce the power consumption of the electronic device in the shooting scene while ensuring the image quality of the photo, the electronic device can control the frame rate of the first camera to a larger frame rate, namely the first frame rate, and control the frame rate of the second camera to a smaller frame rate, namely the second frame rate. In this way, when shooting images subsequently, the electronic device can not only select images with better image quality from enough RAW images historically output by the first camera, but also select images from the RAW images historically output by the second camera to assist in implementing the binocular algorithm to create image effects for the image (for example, using the binocular blur algorithm to blur the image background), which can reduce the power consumption of the electronic device in the shooting scene while ensuring the image quality of the photo.

[0009] In combination with the first aspect, in a possible implementation, the first frame rate is a frame rate at which the first camera and the second camera generate RAW images.

[0010] In this real-time mode, both the first camera and the second camera generate RAW images at the first frame rate. That is, the photosensitive element in the first camera and the photosensitive element in the second camera transmit RAW images to their respective image front ends at the first frame rate. However, the image front end in the first camera will output each received RAW image frame and save it in the cache of the electronic device, while the image front end in the second camera will not output each received RAW image frame. It can output a RAW image frame to the cache of the electronic device at intervals of several frames under the control of the electronic device, so as to reduce the power consumption of the electronic device by the second camera.

[0011] In combination with the first aspect, in a possible implementation, the first frame rate is twice the second frame rate, and controlling the first camera to output the RAW image at the first frame rate includes: controlling the first camera to output each frame of the RAW image generated by the first camera; and controlling the second camera to output the RAW image at the second frame rate includes: controlling the second camera to output one RAW image out of the two RAW images when two RAW images are generated.

[0012] In this embodiment, the image front end in the first camera will output and save each received RAW image in the cache of the electronic device, and the image front end in the second camera will output one of the two RAW images (i.e., output one RAW image every 1 frame) to the cache of the electronic device for each two RAW images generated under the control of the electronic device. It can be understood that although the second camera can also output a RAW image to the cache of the electronic device at intervals of more frames (for example, 2 or 3 frames), the power consumption of the electronic device by the second camera can be reduced to the maximum extent. However, the output of a RAW image at intervals of 1 frame in the second camera can more likely select a RAW image that is closer to the user's shooting time when the image is subsequently output, and the image content of the photo finally obtained based on the RAW image of the historical cache will also be more in line with the user's desired shooting content, which can maximize the image quality while reducing the power consumption of the electronic device by the second camera.

[0013] In combination with the first aspect, in a possible implementation, the method further includes: controlling the first camera to output a YUV image at the first frame rate, processing the first YUV image output by the first camera based on a monocular blur algorithm to obtain a first preview image; and displaying the first preview image in a preview area of ​​a display screen.

[0014] It is understandable that although the blurring effect presented by the monocular blurring algorithm is not as accurate as that presented by the binocular blurring algorithm, the computing power requirements and power consumption of the monocular blurring algorithm on the device are lower than those of the binocular blurring algorithm, and what users need most is the photos taken rather than the preview images displayed during preview. Therefore, in this embodiment, the second camera may not output the YUV image during the preview stage. That is, the second camera can still output the RAW image to the cache of the electronic device, but the second camera will not output the YUV image converted from the RAW image. The images sent in the preview stream are all obtained by processing the YUV image output by the first camera based on the monocular blurring effect, which can reduce the power consumption of the electronic device at the expense of the quality of the blurring effect of the image in the preview stream.

[0015] In combination with the first aspect, in one possible implementation, the method is applied to a zero-second delay shooting mode, and the method also includes: in response to a user operation of a shooting control, saving a first photo, wherein the first photo is obtained by processing at least one frame of a first RAW image output by the first camera and at least one frame of a second RAW image output by the second camera based on a binocular blur algorithm.

[0016] Reducing the output frame rate of the RAW image in the auxiliary camera does not mean reducing the rate at which the image sensor in the auxiliary camera generates the RAW image or the frame rate at which the image sensor transmits the RAW image to the image front end, but rather reducing the frame rate at which the image front end in the auxiliary camera outputs the RAW image and caches it in the cache queue. In this way, after the user subsequently clicks the shooting control to take a photo, the electronic device can still obtain the RAW image cached by the main camera and the RAW image cached by the auxiliary camera from the cache, and use the binocular blur algorithm to blur the two images to ensure the blur effect of the last saved photo.

[0017] In combination with the first aspect, in one possible implementation, the first frame rate is n times the second frame rate, the RAW images output by the first camera are cached in a first cache queue, and the RAW images output by the second camera are cached in a second cache queue, and when the first cache queue and the second cache queue are both full, the RAW images stored in the second cache queue are n times the number of RAW images in the first cache queue, where n is a positive number.

[0018] In order to determine a pair of RAW images with the same generation time from the two RAW images output by the first camera and the second camera during the subsequent shooting and frame selection process, when outputting the RAW images to the cache, the electronic device will add frame numbers to the RAW images. The frame numbers corresponding to the RAW images generated at the same time are the same. When selecting RAW images, the electronic device can determine the required RAW images and the frame numbers corresponding to these RAW images from the RAW images output by the first camera, and then select the RAW images with the same frame numbers from the RAW images output by the other camera based on the frame numbers corresponding to these RAW images. However, due to the limitation of storage space and the fact that the second camera outputs RAW images at intervals, when the user subsequently clicks the shooting control, when the electronic device selects the corresponding RAW images from the RAW images output by the first camera and the RAW images output by the second camera, after the electronic device determines multiple frames of RAW images from the RAW images output by the first camera, the electronic device may not be able to completely find the RAW images with the same frame numbers of the multiple frames of RAW images in the RAW images output by the other camera.

[0019] Therefore, in this embodiment, the electronic device can reduce the length of the second cache queue by half to ensure that any RAW image in the second cache queue has a RAW image with the same frame number as the RAW image in the first cache queue. Then, the electronic device can first determine the frame numbers of the required multiple RAW images from the second cache queue, and then find another multiple RAW images corresponding to the frame numbers of the multiple RAW images from the first cache queue, to avoid the situation where the corresponding RAW image cannot be found and the camera needs to re-frame.

[0020] In combination with the first aspect, in a possible implementation, before saving the first photo, the method also includes: determining at least one frame of the second RAW image from the RAW image output by the second camera; determining at least one frame of the first RAW image from the RAW image output by the first camera based on the frame numbers of the at least two frames of the RAW image, and the number of the at least one frame of the first RAW image is the same as that of the at least one frame of the second RAW image; fusing the at least one frame of the first RAW image based on a multi-frame fusion algorithm to obtain a first image; processing the first key frame and the second key frame based on a binocular blur algorithm to obtain depth of field information of the first key frame, the first key frame being a frame of the at least one frame of the first RAW image with better image quality, and the second key frame being a RAW image with the same frame number as the first key frame in the at least one frame of the second RAW image; blurring the background of the first image based on the depth of field information to obtain the first photo.

[0021] In this embodiment, since any RAW image in the second cache queue has a RAW image with the same frame number as the RAW image in the first cache queue, the electronic device can first determine the at least one second RAW image from the RAW image output by the second camera, and it can also determine the same number of RAW images, that is, the at least one first RAW image, from the RAW image output from the first camera based on the frame numbers of the at least two RAW images. In order to ensure the image quality of the final photo, the electronic device can determine a frame with better image quality in the at least one first RAW image as the first key frame based on the image quality evaluation algorithm, and determine the RAW image with the same generation time as the first key frame in the at least one second RAW image as the second key frame. Based on the first key frame, the information of other images in the at least one first RAW image is superimposed on the first key frame to ensure the clarity of the subsequent photo. Afterwards, the electronic device can obtain the taken photos through the binocular blur algorithm, that is, calculate the depth of field information of each pixel in the first key frame through the first key frame and the second key frame, and accurately determine the foreground and background of the first key frame based on the depth of field information, and then blur the background image. It can improve the clarity of the foreground image while accurately blurring the background image, avoiding the defects of missing background blur or mistaken foreground blur.

[0022] In combination with the first aspect, in one possible implementation, the first camera may be a wide-angle camera, and the second camera may be an ultra-wide-angle camera; or, the first camera may be a telephoto camera, and the second camera may be a wide-angle camera.

[0023] In a second aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method in the first aspect or any possible implementation of the first aspect.

[0024] In a third aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the processors are used to call computer instructions so that the electronic device executes a method as in the first aspect or any possible implementation of the first aspect.

[0025] According to a fourth aspect, a computer program product comprising instructions, when the computer program product is run on an electronic device, enables the electronic device to execute the method according to the first aspect or any possible implementation of the first aspect.

[0026] In a fifth aspect, a computer-readable storage medium is provided, comprising instructions, which, when executed on an electronic device, enable the electronic device to execute a method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of the architecture of a camera system provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of the architecture of a camera system provided in an embodiment of the present application;

[0029] Figure 3 A flowchart of a shooting method provided in an embodiment of the present application;

[0030] Figure 4-Figure 6 Some user interface diagrams provided for embodiments of the present application;

[0031] Figure 7 A schematic diagram of a method for outputting images by an electronic device provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of a RAW image cache queue provided in an embodiment of the present application;

[0033] Fig. 9 An Android system architecture diagram provided for an embodiment of the present application;

[0034] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0036] To facilitate understanding, the relevant terms involved in the embodiments of the present application are first introduced below.

[0037] (1) Multi-device

[0038] A multi-camera device refers to an electronic device with two or more cameras. For example, the electronic device provided in this application may include multiple cameras, which may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, etc., and may also include other types of cameras, which can be placed on the front or back of the electronic device, that is, as the front camera or rear camera of the electronic device.

[0039] Compared with monocular devices (electronic devices with only one camera), multi-eye devices have a wider shooting range and more viewing angles, can capture more comprehensive scenes and multi-dimensional images, and have a wider field of view than traditional monocular devices. Multi-eye devices can synthesize the fields of view of multiple lenses to obtain a wider range of images. Specifically, multi-eye devices can use multiple camera modules to form images separately, and through specific algorithm processing, the images taken by multiple modules are merged into one image to achieve specific imaging needs. Secondly, multi-eye camera devices can achieve depth perception, that is, combined with depth measurement technology, allowing users to better perceive the depth of the scene while recording images. For example, when shooting portraits, multi-eye devices can blur the background based on distance data or make the portrait more realistic while blurring.

[0040] In this application, the image data output by each camera in a multi-eye device can be referred to as a data stream. When a user uses a multi-eye device to frame a target, the data stream output by a camera on the multi-eye device can be used for display (i.e., displayed on the screen of the electronic device for user preview), and the data streams output by other cameras can be used for auxiliary information detection, alignment, ranging and other functions to correct or beautify the image in the above-mentioned data stream for display, and improve the quality of the displayed image. Among them, the camera whose output data stream is used for display can be referred to as the "main camera", and the camera whose output data stream can be used for auxiliary information detection, alignment, and ranging can be referred to as the "auxiliary camera".

[0041] However, the simultaneous output of data streams by multiple cameras will undoubtedly increase the power consumption of electronic devices and occupy a greater amount of computing resources and memory resources of electronic devices.

[0042] (2) Background blur

[0043] Background blur, also known as depth of field effect, is a camera technology that makes the depth of field shallower, focuses the lens on the main body of the picture, and the background appears blurred. It is most commonly used in the portrait mode provided by photo software. When shooting an object in portrait mode, the electronic device can automatically blur the background to highlight the subject.

[0044] Currently, the background blurring technologies commonly used in electronic devices include monocular blurring and binocular blurring.

[0045] Monocular blur refers to the technology that achieves blur effect through only one camera. For monocular blur technology, electronic devices can control the degree and range of blur by adjusting the focal length and aperture size of the camera. It can be understood that when the focal length of the camera is far, the aperture is small, the depth of field is large, and the whole picture will be clearer; when the camera is close, the aperture is large, the depth of field is small, only the objects near the focus will be clear, and other parts will appear blurred. In this way, electronic devices can achieve the blur effect by adjusting the focal length and aperture of the camera to highlight the subject in the image and blur the background.

[0046] Binocular blurring requires the use of two cameras to achieve blurring effects, so it needs to be applied to multi-eye devices. In addition, due to the limitations of the triangulation principle, the two camera modules need to be calibrated so that the imaging planes of the two are on the same plane and the pixels are aligned. In other words, the two cameras used for shooting need to be on the same side of the electronic device, that is, both cameras need to be the front cameras of the electronic device or the rear cameras of the electronic device. In a dual-camera system, one camera is responsible for shooting the subject, and its output image data stream can be used for display; while the other camera is responsible for measuring the depth of field information of each point, that is, the electronic device can determine the distance from each object in the scene to the electronic device (camera) through the image taken by the camera, and separate the foreground and background according to the information, and then use the blurring algorithm to blur the background. Compared with monocular blurring, binocular blurring can be more accurate and delicate for the blurring of the subject, thereby simulating the visual effect of the human eye to a greater extent, making the blurring effect more natural.

[0047] (3) RAW images and YUV images

[0048] RAW images are the raw data that the image sensor converts the captured light source signal into a digital signal. It is the data format output by the sensor. An image sensor can be called a photosensitive element. It is a device that converts an optical image into an electronic signal, such as a charge coupled device (CCD) image sensor or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In the field of image technology, the RAW format is an unprocessed, uncompressed format, that is, the original image encoding data (digital negative). Common suffixes for the RAW format include .ARW, .SRF, .SR2, .crw, .cr2, .cr3, etc.

[0049] YUV is a digital image format that separates image brightness (Y) and color (U, V), and is often used in the fields of video encoding and digital image processing. In fact, YUV is a color encoding method that separates brightness information (Y) from color information (UV), and can display a complete image without UV information, but the image is black and white.

[0050] It should be understood that the RAW output of the sensor Figure 1 Generally, it will not be directly displayed on the display screen of electronic devices, because the human eye cannot directly obtain scene information directly from the RAW image. This is because CMOS sensors are generally single-channel image sensors, and each pixel only records one channel signal. Therefore, the RAW image output by the sensor needs to undergo a series of operations including white balance correction, color space conversion, and tone mapping before the YUV image can be finally displayed on the display screen of the electronic device.

[0051] (4) Zero shutter lag (ZSL)

[0052] In daily life, when taking photos with a mobile phone, there is often a delay experience, that is, after the user clicks the shooting control on the mobile phone. It usually takes a while for the electronic device to output the captured image to the user. This is because in non-ZSL mode, the electronic device starts previewing and outputs a preview frame; when the user presses the shutter, the preview stream will be stopped, and the electronic device needs to perform a series of actions, including AF focusing, preparing to take photos, exposure, and photo stream data callback. At this time, the image obtained is a RAW image. After that, the RAW image is encoded to obtain a JPEG image and save it to the set storage area. At this time, the photo is completed and the preview is restarted. In other words, in non-ZSL mode, the preview is stopped when taking photos, and the RAW image used in the preview stream is not stored in the electronic device. Only after the user presses the shutter, the electronic device will store several frames of RAW images and process them to obtain images that can be browsed by the user (such as JPEG format images). For example, when a user takes a picture of a jumping target, the user wants to capture the moment the target jumps and press the shutter at that moment the target jumps. However, in non-ZSL mode, the electronic device can only use the RAW image captured by the camera after the user presses the shutter to process the final photo. After the user presses the shutter, the target may have already jumped and returned to the ground. Therefore, the RAW image captured by the camera after the user presses the shutter is not the image captured at the moment the target jumps. The final photo may not show the picture of the target jumping, but the picture of the target landing. That is to say, in non-ZSL shooting mode, there is a certain delay between the picture taken by the user and the picture the user actually wants to take.

[0053] ZSL is a camera mode that is developed to eliminate this delay and to provide a "shoot and see" experience. In ZSL mode, when the electronic device starts previewing, the raw image generated by the sensor will be stored in the cache. When the user presses the shutter, the device system will calculate the actual shooting time, find the RAW image of the corresponding frame stored in the cache, and then use the RAW image to encode, obtain the JPEG image and save it to the set storage area. In other words, in ZSL mode, the preview can not stop when the user clicks the shutter to shoot, and the RAW image corresponding to the most recently used image in the preview stream will be stored in the cache. After the user presses the shutter, the system can calculate the delay time (that is, the time difference between the time the user actually wants to shoot and the time the user actually presses the shutter, that is, the user's reaction time), and then recognize a certain image frame as the real-time frame of the photo, and process it to obtain an image that can be browsed by the user (such as an image in JPEG format).

[0054] (5) Camera structure

[0055] In the present application, each camera of the electronic device may include an image sensor and an image processor (image signal processing, ISP). Among them, the sensor is the photosensitive element in the above description, which can convert photons into electronic signals, and convert them into digital signals through an amplification circuit and an analog-to-electric conversion circuit, that is, a RAW image. The ISP may include an image front end (image front end, IFE). During the operation of the camera, the RAW image output by the sensor will first be transmitted to the IFE, and the IFE can perform color correction, downsampling, de-mosaicing and other processing on the RAW image to obtain a YUV image; the IFE will further output the YUV image to the image processing engine (image processing engine, IPE), and the IPE will perform hardware noise reduction, cropping, noise reduction, color processing, detail enhancement and other image processing on the YUV image to obtain the final YUV image for display (the image is an image for user preview, not a photo obtained after the user clicks the shooting control). If the current shooting mode of the electronic device is ZSL mode, the IFE will further store the received RAW image in the cache. When the user clicks the shooting control, the electronic device can calculate the actual shooting time, find the RAW image of the corresponding frame stored in the cache, and then use the RAW image to encode it to obtain a JPEG image and save it to the set storage area. The JPEG image is the photo obtained by taking the photo.

[0056] It should be understood that in addition to the above-mentioned components, each camera in the present application may also include other components, such as lenses, focus motors, filters, etc., and the present application does not limit this.

[0057] (6) Frame rate

[0058] Frame rate is the frequency (rate) at which a bitmap image in units of frames appears continuously on a display. In an embodiment of the present application, the frame rate can represent the number of images output or received by a camera (camera element) per second. For example, it can be used to represent the number of RAW images stored in the cache output by the IFE in the camera per second, and can also be used to represent the number of YUV images output by the IFE or IPE in the camera per second. Specifically, assuming that the first camera in the multi-eye device saves a RAW image in the cache buffer every 0.02 seconds, and the second camera saves a RAW image in the cache buffer every 0.04 seconds, it can be considered that the frame rate of the RAW image output by the first camera is twice that of the RAW image output by the second camera.

[0059] (7) Application layer, application framework layer, hardware abstraction layer

[0060] The application layer (Application), application framework layer (Android Framework), and hardware abstraction layer (Hardware Abstraction Layer) are the four layers in the Android system architecture. Among them:

[0061] The application layer includes system built-in applications and non-system-level applications, which are usually developed based on the Java language and are mainly responsible for direct interaction with users. For example, when a user operates an electronic device running on the Android system and opens a photo-taking application (such as a camera), the application layer can respond to the user operation, send the request to the application framework layer through the Camera Api v2 standard interface, and wait for the application framework layer to return the processing results, which include image data and overall camera system status parameters, and then feed back the results to the user in a certain way, such as arranging the position and drawing the content of each layer in the image returned by the application framework layer through the SurfaceFlinger module, so that the image is finally displayed on the screen of the electronic device.

[0062] The application framework layer is the foundation of Android application development. This layer is written in Java code and provides developers with the APIs they need to develop applications. Many core applications also use this layer to implement their core functions. This layer simplifies the reuse of components. Developers can directly use the components it provides to develop applications quickly, or they can implement personalized expansion through inheritance. Specifically, the application framework layer can include multiple parts such as activity manager, window manager, content provider, view system, package manager, phone manager, resource manager, location manager, notification manager, etc.

[0063] The hardware abstraction layer encapsulates the underlying hardware driver and provides a general interface for calling the driver to the application framework layer. As long as the manufacturer implements the corresponding interface according to the specification and stores it in a specific directory in the form of a shared library, the upper layer only needs to load the shared library and find the pointer of the device corresponding to the corresponding module. After obtaining the pointer of the entire device, the underlying hardware can be operated. For example, the aforementioned application framework layer can send the frame request instruction received from the application layer to the hardware abstraction layer, so that the sensor in the hardware abstraction layer generates a frame of RAW image according to the instruction, and passes the RAW image to IFE. After being converted into a YUV image by IFE, the YUV image is further passed to IPE for processing. The final image can be returned to the application layer through the application framework layer for display.

[0064] In order to pursue better image quality and create various image effects for images, most electronic devices nowadays are usually equipped with multiple cameras. For example, the rear camera of a mobile phone may include multiple cameras such as a wide-angle camera, an ultra-wide-angle camera, and a telephoto camera. In many shooting scenes, multi-eye devices use two cameras at the same time to shoot. For example, in portrait mode, a multi-eye device can use one camera to shoot the entire picture, and the data stream output by the camera will be used for display; at the same time, the multi-eye device can also obtain depth of field information through another camera, blur the background outside the subject in the image, and then highlight the subject in the image.

[0065] Figure 1 The working mode of the camera system 10 in the multi-eye electronic device in the ZSL mode and the portrait mode is shown.

[0066] like Figure 1 As shown, the camera system 10 may include a camera 101, a camera 102, and an image processing engine 103. The camera 101 may include an image sensor 1011 and an image front end 1012, and the camera 102 may include an image sensor 1021 and an image front end 1022. In the embodiment of the present application, the camera 101 may be referred to as a main camera, and the camera 102 may be referred to as an auxiliary camera. In some embodiments, the image processing engine 103 may work in conjunction with the camera 101 and the camera 102 respectively under the control of software code, and the processing logic when the image processing engine 103 works in conjunction with the camera 101 and the processing logic when the image processing engine 103 works in conjunction with the camera 102 may be different, and the two may be performed simultaneously without affecting each other.

[0067] After the user opens a shooting application (such as a camera), the camera 101 and the camera 102 can output images at the same time.

[0068] Specifically, the image sensor 1011 of the camera 101 can collect light signals at a certain collection rate, convert the light signals into RAW images, and send the generated RAW images to the image front end 1012. Since the electronic device needs to use the RAW images stored in the cache to generate photos in the ZSL shooting mode, after the image front end 1012 receives the RAW images transmitted by the image sensor 1011, on the one hand, the image front end 1012 will perform color correction, de-mosaicing and other processing on the received RAW images, convert the RAW images into YUV images, and transmit the YUV images to the image processing engine 103 at a frame rate of fps1 (that is, the image front end 1012 sends fps1 frames of RAW images to the image processing engine 103 per second, the same below); on the other hand, the image front end 1012 will also output the RAW images to the cache of the electronic device at a frame rate of fps1, that is Figure 1After receiving the YUV image sent by the image front end 1012, the image processing engine 103 will further perform noise reduction, cropping and other processing on the YUV image to obtain the main YUV image for display.

[0069] At the same time, the image sensor 1021 of the camera 102 can also collect light signals at a certain collection rate, and convert the collected light signals into RAW images, and send the generated RAW images to the image front end 1022. Similarly, after the image front end 1022 receives the RAW image transmitted by the image sensor 1021, the image front end 1022 will process and convert the received RAW image into a YUV image, and also send the YUV image to the image processing engine 103 at a frame rate of fps1; on the other hand, the image front end 1022 will output the RAW image to the cache of the electronic device at a frame rate of fps1, that is, Figure 1 After receiving the YUV image sent by the image front end 1022, the image processing engine 103 will further perform noise reduction, cropping and other processing on the YUV image to obtain an auxiliary YUV image for assisting the aforementioned main YUV image in sending and displaying.

[0070] It should be noted here that the number of RAW images sent by the image sensor 1011 to the image front end 1012 per second and the number of RAW images sent by the image sensor 1021 to the image front end 1022 per second can be the same or different, which can be specifically determined by the performance of the camera 101 and the camera 102 and / or the program code logic set when the electronic device leaves the factory. However, the image front end 1012 and the image front end 1022 will send YUV images to the image processing engine 103 and the image processing engine 103 respectively at the same frame rate fps1, and output the RAW images to the cache at the same frame rate fps1.

[0071] In the real-time preview stage, the electronic device can further process the main YUV image output by the image processing engine 103 through the binocular blur algorithm to obtain an image with a blur effect that is finally displayed on the screen for the user to preview. Figure 1 The depth of field information of each pixel in the main YUV image is calculated, and then the foreground and background in the main YUV image are determined, and the background of the main YUV image is blurred. The YUV image obtained after the blurring process can be transmitted to the screen of the electronic device for the user to preview.

[0072] During the shooting stage, that is, after the user clicks the shooting control (shutter), the electronic device can select several frames of main RAW images with historical cache from the main RAW image cache queue according to the shooting time (that is, the time when the user clicks the shooting control), and based on the selected main RAW images, select RAW images with the same number of frames with the same main RAW image generation time from the auxiliary RAW image cache queue, and then blur them and fuse multiple frames to obtain the final photo stored in the electronic device.

[0073] In some embodiments, the electronic device may also select several frames of historically cached auxiliary RAW images from the auxiliary RAW image cache queue based on the photo taking time (i.e., the time when the user clicks the shooting control), and based on the selected auxiliary RAW images, select RAW images with the same number of frames as the main RAW image generation time from the main RAW image cache queue, and then blur them and fuse multiple frames to obtain the final photo stored in the electronic device.

[0074] Optionally, the electronic device can determine three RAW images P1, P2 and P3 from the above-mentioned main RAW image cache queue, and determine the corresponding three RAW images P1', P2' and P3' from the above-mentioned auxiliary RAW image cache queue based on P1, P2 and P3, wherein the numbers of the caches used to store P1, P2, P3 are respectively the same as the numbers of the caches used to store P1', P2', P3' (that is, the image sensor 1011 and the image sensor 1021 generate P1 and P1' respectively for the exposure processing of the shooting scene at the same time, and generate P2 and P2' respectively for the exposure processing of the shooting scene at another time, and so on). The electronic device can calculate the depth of field information of each pixel point in P1 based on P1, through P1 and P1', and determine the foreground and background of P1 based on the depth of field information, and then blur the background in P1 to obtain a frame of image with a blurred effect; similarly, the electronic device can obtain another two images with a blurred effect based on P2 and P2', P3 and P3'. Afterwards, the electronic device can use one of the images with the blur effect as a key frame, and superimpose the information in the other two images with the blur effect onto the key frame through a multi-frame fusion algorithm. The final image can be stored in the electronic device as a photo.

[0075] Optionally, in some embodiments, after the user clicks the shooting control, both the image sensor 1011 and the image sensor 1021 can continue to output RAW images and continue to transmit them to the image front end 1012 and the image front end 1022 respectively. The image front end 1012 and the image front end 1022 can also continue to output the received RAW images and store them in the above-mentioned main RAW image cache queue and the above-mentioned auxiliary RAW image cache queue respectively. Afterwards, the electronic device can determine six RAW images P1, P2, P3, P4, P5, and P6 from the above-mentioned main RAW image cache queue (wherein P1, P2, and P3 are RAW images cached before the user clicks the shooting control, and P4, P5, and P6 are RAW images cached after the user clicks the shooting control), and based on P1, P2, P3, P4, P5, and P6, determine the corresponding six RAW images P1', P2', P3', P4', P5', and P6' from the above-mentioned auxiliary RAW image cache queue (wherein P1', P2', and P3' are RAW images cached before the user clicks the shooting control, and P4', P5', and P6' are RAW images cached after the user clicks the shooting control); afterwards, The electronic device can determine the key frame from the six RAW images of P1, P2, P3, P4, P5, and P6 through image quality evaluation (here it is assumed that the image quality of P1 is the best and is determined as the key frame), and use the remaining frames (i.e., P2-P6) as reference frames, first superimpose the information in all reference frames on the key frame through a multi-frame fusion algorithm, and obtain a frame of image P7 fused from multiple frames, and then calculate the depth of field information of each pixel in P1 through P1 and P1' (i.e., the reference frame and the auxiliary RAW image corresponding to the reference frame), and determine the foreground and background of P7 based on the depth of field information, and then blur the background in P7 to obtain an image P8 with a blurred effect, which is stored in the electronic device as the photograph obtained by the shooting. Optionally, after the user clicks the shooting control, the electronic device can use other methods to process the RAW images cached in the history to obtain the above-mentioned photos, which can be specifically determined by the algorithm logic of the multi-frame fusion algorithm and the binocular blur algorithm adopted by the electronic device, and this application does not limit this. However, in order to ensure the background blur effect of the photo and there is no delay between the photo image and the time when the user takes the photo, the historical cached RAW image obtained by the electronic device (that is, the RAW image stored before the user clicks the shooting control) must contain both the RAW image historically output by camera 101 and the RAW image historically output by camera 102.

[0076] Combined with the above description, it can be seen that in ZSL shooting mode, in order to achieve a more accurate background blur effect for the image in portrait mode, the two cameras contained in the multi-eye electronic device will use binocular blur technology to blur the image regardless of previewing and shooting. In addition, when previewing, both cameras need to output YUV images at the same frame rate for users to preview, and store RAW images in the cache queue at the same frame rate, so that the electronic device can select frames and output images with zero delay after the subsequent user clicks the shooting control. However, for multi-eye electronic devices, the continuous collaboration of multiple cameras will increase the power consumption of the electronic device, especially when using multiple cameras to shoot portraits, the excessively long links involved in the portrait algorithm will place a greater burden on the memory resources and computing power of the electronic device.

[0077] In response to the above problems, the present application provides a shooting method, which reduces the working intensity of the auxiliary camera by reducing the frame rate of the auxiliary camera in the preview stage while ensuring the frame rate of the main camera, thereby reducing the power consumption of the electronic device during shooting and the burden on the computing power and memory of the electronic device.

[0078] It should be noted in advance that although the blurring effect presented by the monocular blurring algorithm is not as accurate as that presented by the binocular blurring algorithm, the monocular blurring algorithm has lower computing power requirements and power consumption for the device than the binocular blurring algorithm, and the user most needs to obtain the photos taken rather than the preview images displayed during preview. Therefore, in this method, the auxiliary camera can reduce the frame rate of the image output during the preview stage in two aspects: 1. During the preview stage, the auxiliary camera does not output the YUV image. That is, the image sensor of the auxiliary camera can still output the RAW image to the image front end, but the image front end will not output the converted YUV image to the image processing engine, and the image processing engine (auxiliary camera) will not output the YUV image. The images sent in the preview stream are all processed by the monocular blurring algorithm on the YUV image output by the main camera, that is, the quality of the image in the preview stream is sacrificed to reduce the power consumption of the electronic device. 2. In the preview stage, the auxiliary camera reduces the output frame rate of the RAW image. This does not mean reducing the rate at which the image sensor in the auxiliary camera generates the RAW image or the frame rate at which the image sensor transmits the RAW image to the image front end, but rather reducing the frame rate at which the image front end in the auxiliary camera outputs the RAW image and caches it in the cache queue. In this way, after the user clicks the shooting control to take a photo, the electronic device can still obtain the RAW image cached by the main camera and the RAW image cached by the auxiliary camera from the cache, and use the binocular blur algorithm to blur the two images to ensure the blur effect of the last saved photo. However, for the shooting scene in ZSL mode, if the auxiliary camera does not output and cache the RAW image at all in the preview stage, then the electronic device cannot get the ZSL frame cached historically by the auxiliary camera in the photo shooting stage, and the electronic device cannot use the binocular blur algorithm to obtain an image with a better blur effect. Therefore, in ZSL mode, the auxiliary camera in the preview stage can output the RAW image generated by the image sensor in an intermittent frame output mode, but it cannot completely output the RAW image in the cache.

[0079] Figure 2 The specific working mode of the camera system of the electronic device provided by the present application in the ZSL mode is shown. Figure 2 As shown, the camera system 20 may include a camera 201, a camera 202, and an image processing engine 203. The camera 201 may include an image sensor 2011 and an image front end 2012, and the camera 202 may include an image sensor 2021 and an image front end 2022. In some embodiments, the image processing engine 203 may work in coordination with the camera 201 and the camera 202 respectively under the control of software code, and the processing logic when the image processing engine 203 and the camera 201 work in coordination with each other may be different, and the two may be performed simultaneously without affecting each other.

[0080] Among them, camera 201 can be called a main camera, and camera 202 can be called an auxiliary camera. Camera 201 and camera 202 are both front cameras, or camera 201 and camera 202 are both rear cameras. This application does not limit the types of camera 202 and camera 201. Specifically, camera 201 can be a wide-angle camera, and camera 202 can be an ultra-wide-angle camera; or, camera 201 can be a telephoto camera, and camera 202 can be a wide-angle camera. After the user opens a shooting application (such as a camera), camera 201 and camera 202 can work in different ways.

[0081] Specifically, the image sensor 2011 of the camera 201 collects light signals at a certain collection rate, converts the light signals into RAW images, and transmits the generated RAW images to the image front end 2012. After receiving the RAW images transmitted by the image sensor 2011, the image front end 2012 will perform color correction, de-mosaicing and other processing on the received RAW images, convert the RAW images into YUV images, and send the YUV images to the image processing engine 203 at a frame rate of fps2. The image processing engine 203 performs noise reduction, cropping and other processing on the YUV images to obtain the main YUV images for display; in addition, the image front end 2012 will output the RAW images to the cache of the electronic device at a frame rate of fps2, that is, Figure 2 In the main RAW image cache queue.

[0082] The image sensor 2021 of the camera 202 can also collect light signals at a certain collection rate, convert the collected light signals into RAW images, and send the generated RAW images to the image front end 2022 at a frame rate of fps6. After the image front end 2022 receives the RAW image transmitted by the image sensor, the image front end 2022 will not further send the received YUV image to the image processing engine 203 (at this time, the YUV image transmission channel between the image front end 2022 and the image processing engine 203 is closed); however, the image front end 2022 will output the RAW image to the cache of the electronic device at a frame rate of fps3, that is, Figure 2 In the auxiliary RAW image cache queue.

[0083] Specifically, the electronic device can control the image front end 2022 in the camera 202 to output the RAW image to the auxiliary RAW image cache queue at a fixed frame interval; for example, the image front end 2022 can cache the RAW image transmitted by the image sensor 2021 to the auxiliary RAW image cache queue in a manner of caching one frame every two frames under the control of the electronic device. For example, when the image sensor 2021 transmits the generated first frame of RAW image to the image front end 2022, the image front end 2022 can output the first frame of image to the auxiliary RAW image cache queue; when the image sensor 2021 transmits the generated second frame of RAW image to the image front end 2022, the image front end 2022 will not output the second frame of image to the auxiliary RAW image cache queue; when the image sensor 2021 transmits the generated third frame of RAW image to the image front end 2022, the image front end 2022 will not output the third frame of image to the auxiliary RAW image cache queue, and so on.

[0084] Optionally, the value of fps3 can be 1 / 2 of the value of fps2. Alternatively, the value of fps2 can also be 1 / 3 of the value of fps2, or other values ​​smaller than the value of fps2, which is not limited in this application. It can be understood that Figure 2 The number of RAW images in the main RAW image cache queue N will be greater than Figure 2 The number of RAW images in the RAW image cache queue of the auxiliary path is n. For example, if the value of fps3 is 1 / 2 of the value of fps2, in the same duration, the above N will be equal to 2n.

[0085] Similarly, the number of RAW images sent by image sensor 2011 to image front end 2012 per second and the number of RAW images sent by image sensor 2021 to image front end 2022 per second may be the same or different, which may be specifically determined by the performance of camera 201 and camera 202 and / or the program code logic set when the electronic device leaves the factory.

[0086] In the real-time preview stage, the electronic device can further process the main YUV image output by the image processing engine 203 through a monocular blur algorithm to obtain an image with a blur effect that is finally displayed on the screen for the user to preview. Specifically, the electronic device can identify the subject in the main YUV image through an AI algorithm, blur the background, and transmit the YUV image obtained after the blur processing to the screen of the electronic device for the user to preview.

[0087] During the shooting stage, the electronic device can select several frames of historically cached RAW images from the main road RAW image cache queue and the secondary road RAW image cache queue respectively according to the shooting time (i.e., the time when the user clicks the shooting control), and then perform blurring processing and fuse multiple frames of images with blurring effects to obtain the photo finally stored in the electronic device. For details, please refer to the relevant descriptions of Figure 1 and the relevant descriptions of the subsequent embodiments, which will not be elaborated here.

[0088] Of course, in some embodiments, if a preview image with a better blurring effect needs to be displayed for the user during the real-time preview stage, the electronic device can also control the image front end 2022 in the secondary road camera (i.e., camera 202) to send the YUV image to the image processing engine 203 during the preview stage, and at the same time reduce the frame rate of outputting the RAW image to the secondary road RAW image cache queue. In this way, the electronic device can provide a preview image with a better blurring effect for the user during the preview stage, and can also reduce the working intensity of the secondary road camera to a certain extent, thereby reducing the power consumption of the electronic device.

[0089] Combined with the relevant descriptions of the imaging system 20 above, the shooting method provided by this application will be introduced next. Please refer to Figure 3 .

[0090] Figure 3 is a flowchart of a shooting method provided by an embodiment of this application. Figure 3 Taking the preview stream adopting a monocular blurring algorithm and the secondary road camera caching one frame of RAW image every two frames as an example, the shooting method provided by this application is illustrated. As Figure 3 shown, in the shooting method provided by an embodiment of this application, the first acquisition module and the first IPE module can be included in the first camera, and the first camera can be the main road camera in this method; the second acquisition module and the second IPE module can be included in the second camera, and the second camera can be the secondary road camera in this method. Among them, the first camera can be the aforementioned camera 201, and the second camera can be the aforementioned camera 202. This application does not limit the specific types of the first camera and the second camera. Specifically, the first camera can be a wide-angle camera, and the second camera can be an ultra-wide-angle camera; or, the first camera can be a telephoto camera, and the second camera can be a wide-angle camera.

[0091] When implementing this method, the camera application of the application layer can load the portrait mode in response to the user's operation of opening the camera application. After the portrait mode is loaded, the user can start the portrait mode by touching the portrait mode icon. Then, the hardware abstraction layer can identify the shooting scene and report it to the shooting control module of the application layer. The shooting control module can adjust the shooting parameters and shooting mode in the portrait mode and send it back to the image acquisition module of the hardware abstraction layer. Finally, the image acquisition module can capture images according to the adjusted shooting parameters and shooting mode. The first IFE module, the IPE module, the second IFE module and the IPE module can determine the image processing algorithm used according to the identified shooting scene, and use the image processing algorithm to process the collected image, and the image data stream obtained after processing can be encoded to obtain an image file. The preview display module can also obtain the image data stream obtained after processing for preview display.

[0092] In this method, the IPE module can work in conjunction with the first camera and the second camera respectively under the control of the software code. The processing logic when the IPE module works in conjunction with the camera 101 and the processing logic when the image processing engine 103 works in conjunction with the camera 102 can be different, and the two can be carried out simultaneously without affecting each other.

[0093] Understandably, in Figure 4 In the Android framework, data (including instructions, information, and images) between the application layer and the hardware abstraction layer can be transmitted via the application framework layer (Android Framework). For the sake of convenience, Figure 4 The information transmission process of the application framework layer is omitted. The shooting method may include but is not limited to the following steps:

[0094] S101: A user starts a camera application.

[0095] In the embodiment of the present application, the user can start the camera application by operating the application icon of the camera application, such as touching the application icon. Figure 4 Detailed description of (A) in .

[0096] S102: The mode loading module loads the mode.

[0097] When the camera application is started, the mode loading module can query the hardware abstraction layer for the mode. In the embodiment of the present application, the hardware abstraction layer can provide a portrait mode for the camera application. That is, in the portrait mode, in the hardware abstraction layer, the image acquisition module, the first IFE module and the IPE module, the second IFE module and the second IFE module can be started to perform their respective functions.

[0098] In the embodiment of the present application, the hardware abstraction layer can also provide other modes for the camera application, such as normal mode, night scene mode and video recording mode, etc., which is not limited in the embodiment of the present application.

[0099] Specifically, the mode loading module may query the hardware abstraction layer for the mode. In response to the query of the mode loading module, the hardware abstraction layer may feedback to the mode loading module the mode provided by the hardware abstraction layer for the camera application, for example, the provided modes include: portrait mode, normal mode, night scene mode and video mode.

[0100] Among them, the loaded modes include the portrait mode. During the loading process, the mode loading module also initializes the modules corresponding to each mode in the application layer and the hardware abstraction layer. After initialization, the electronic device can display the icon corresponding to each mode. For details, please refer to Figure 4 and Figure 5 . After initialization, in response to a user's touch operation on an icon corresponding to the portrait mode, the shooting control module may notify the image acquisition module, the first IFE module and the IPE module, the second IFE module and the second IFE module in the hardware abstraction layer to start to perform their respective functions. After initialization, other modes are similar to the portrait mode, and the corresponding modules in the hardware abstraction layer may be started in response to a user's touch operation on an icon corresponding to the mode.

[0101] The following describes the user interface involved in loading the portrait mode process. Figure 4 , Figure 4 Schematic diagram of a human-computer interaction interface provided by an embodiment of the present application. Figure 4 As shown in (A) in FIG. 1 , the electronic device can display a user interface 11. The user interface 11 includes a calendar widget 111, a weather widget 112, an application icon 113, a status bar 114, and a navigation bar 115. The application icon 113 may include a gallery icon and a camera icon 1131, etc., and may also include icons of other applications, which are not limited in the embodiments of the present application. Any application icon can be used to respond to a user operation, such as a touch operation, so that the electronic device starts the application corresponding to the icon.

[0102] The user can start the camera application by touching the camera icon 1131. Figure 4 As shown in (A) in FIG. 1 , in response to the user's touch operation on the camera icon, the mode loading module executes step S102. After the mode loading module completes loading the mode, the electronic device can display the icon corresponding to each mode.

[0103] Exemplarily, the loaded modes include night scene mode, portrait mode, photo mode, video mode, etc. Figure 4 As shown in (B) in FIG. 1 , the electronic device may display a camera application interface 21. Figure 4 As shown in (B) in FIG. 1 , the camera application interface 21 may also include a captured image display control 212, a capture control 213, a camera switch control 214, a viewfinder 215, a focus control 216A, a setting control 216B, and a filter control 216C. Among them:

[0104] The shooting control 213 is used to respond to user operations, shoot and save the shot photos.

[0105] The captured image echo control 212 is used for the user to view the captured pictures and videos.

[0106] The camera switching control 214 is used to switch the camera for collecting images between the front camera and the rear camera.

[0107] The viewfinder 215 is used for real-time preview display of the collected pictures.

[0108] The focus control 216A is used to adjust the focus of the camera.

[0109] The setting control 216B is used to set various parameters when acquiring images.

[0110] The filter control 216C is used to select the filter effect when shooting.

[0111] The camera application interface 21 may include icons 211 corresponding to loaded modes. Icons 211 may include night scene mode icons 211A, portrait mode icons 211B, photo mode icons 211C, video mode icons 211D, and more icons 211E. The shooting control module may start the mode corresponding to the icon in response to a user's touch operation on any of the icons 211. In an embodiment of the present application, the electronic device may open the camera application in response to a user operation, and then display the camera application interface 21 on the display screen. The user may operate any of the above-mentioned mode icons, such as touching to start the corresponding shooting mode, and the electronic device starts the corresponding module in the hardware abstraction layer.

[0112] S103: The user switches to portrait mode.

[0113] like Figure 4 As shown in (B) in FIG. 2 , the user can touch the portrait mode icon 211B on the camera application interface 21 to switch to the portrait mode.

[0114] S104: The shooting control module is started.

[0115] S105: The preview display module is started.

[0116] In response to the user's touch operation on the portrait mode icon 211B, both the shooting control module and the preview display module are started.

[0117] S106: The shooting control module sends a request instruction for starting the shooting mode to the first acquisition module and the second acquisition module of the hardware abstraction layer.

[0118] After the shooting control module and the preview display module are started, the shooting control module can enable the modules related to the portrait mode in the hardware abstraction layer, such as the image acquisition module. Specifically, the shooting control module can enable the first acquisition module and the second acquisition module in the hardware abstraction layer to start by sending a preview frame request to the hardware abstraction layer, and the first acquisition module and the second acquisition module can start the image acquisition work.

[0119] In a possible implementation, the shooting control module and the preview display module may have been started in step S102, that is, in response to the user starting the camera application, the shooting control module and the preview display module are started. The shooting control module can be used for shooting control in various modes. The preview display module can be used for preview display in various modes.

[0120] S107: The first acquisition module and the second acquisition module generate a RAW image according to preset shooting parameters.

[0121] Among them, the preset shooting parameters may include shooting parameters that may include any one or more of the following: shutter, exposure time, aperture value, exposure value, ISO. The shooting control module may set a shooting parameter and shooting method corresponding to each shooting mode. Exemplarily, the shooting parameter set by the shooting control module in portrait mode may be a first shooting parameter, and the shooting parameter set by the shooting control module in night scene mode may be a second shooting parameter. After the electronic device starts the portrait mode, the shooting control module may send the shooting parameters corresponding to the portrait mode to the image acquisition module together with the enable startup instruction, so that the image acquisition module generates a RAW image according to the shooting parameters corresponding to the portrait mode.

[0122] Specifically, the first acquisition module and the second acquisition module in the image acquisition module can acquire light signals in the shooting scene at the same acquisition rate, and convert the acquired light signals into RAW images in the form of digital signals at the same generation rate.

[0123] The first acquisition module and the second acquisition module can transmit the RAW image to the first IFE module and the second IFE module at the same transmission rate. In the subsequent process, the first IFE module and the second IFE module will process and transmit the received RAW image in different ways, thereby creating a background blur effect for preview stream images and taken photos while reducing the power consumption of the electronic device.

[0124] It should be noted in advance that in some embodiments, the background blur effect of the electronic device on the image in the portrait mode can be turned on or off through corresponding operations. Figure 5 Take this as an example to illustrate. Figure 4 After the touch operation shown in (B) in FIG. 1 switches the shooting mode to the portrait mode, the electronic device may display the following Figure 5 The user interface 31 shown in (A) in FIG. Figure 5 As shown in (A), the user interface 31 is a shooting interface of the portrait mode in the camera application, and the user interface 31 may include a viewfinder 311, a background blur control 312, and a skin beautification control 313. Among them:

[0125] The viewfinder 311 is used for real-time preview display of the collected pictures. A face image 3111 and a tree image 3112 are displayed in the viewfinder 311. When the background is blurred, the electronic device can determine the face image 3111 as the foreground (subject) and the tree image 3112 as the background.

[0126] The background blur control 312 is used to blur the background of the image to highlight the foreground (subject) in the image.

[0127] The skin beautification control 313 is used to beautify the appearance of the subject in the image.

[0128] like Figure 5 As shown in (A) in FIG. 1 , when the electronic device just starts the portrait mode, the background blur control may be in a closed state by default. Since the electronic device may not blur the background of the image in the preview stream at this time, the electronic device may not output the preview image according to the processing logic of the subsequent steps S108 to S127. In response to the touch operation of the background blur control 312, the electronic device will display the preview image with blurred background for the user in the viewfinder according to the processing logic of the subsequent steps S108 to S127, and display the following Figure 5 The user interface 41 shown in (B) in FIG.

[0129] It can be seen from the picture displayed in the preview box 411 in the user interface 41 that the face image 4111 as the foreground has a relatively clear overall picture and outline, while the background image such as the tree image 4112 has been blurred, so the face image 4111 is more prominent in the preview box 411.

[0130] Optionally, in some embodiments, when the electronic device just starts the portrait mode, the background blur control can also be turned on by default. Figure 5After the touch operation shown in (B) in [the figure] switches the shooting mode to the portrait mode, the electronic device can directly blur the background of the image in the preview stream according to the processing logic of subsequent steps S108 - step S127, and the electronic device can directly display the user interface 41 shown in (B) in Figure 5 [the figure].

[0131] S108. The image acquisition module sends the first RAW image to the first IFE module.

[0132] S109. The first IFE module outputs the first RAW image.

[0133] S110. The first IFE module converts the first RAW image into the first YUV image.

[0134] S111. The first IFE module sends the first YUV image to the IPE module.

[0135] It can be understood that steps S108 - step S111 can be executed by the aforementioned first camera (main path camera, including the aforementioned first acquisition module, first IFE module, and IPE module). Among them, the first RAW image is generated by the first acquisition module included in the above image acquisition module and sent to the IPE module.

[0136] Specifically, the first acquisition module can acquire the optical signal in the shooting scene at a certain acquisition rate. After converting the optical signal into the above first RAW image, the first acquisition module can send the first RAW image to the first IFE module. Since the electronic device needs to use the RAW image stored in the cache to generate a photo in the ZSL shooting mode, and the first camera is also responsible for outputting the preview image for the user to preview, therefore, for the first RAW image conveyed by the first acquisition module, after receiving the first RAW image, the first IFE module will process the first RAW image in two aspects: on the one hand, the first IFE module will perform color correction, demosaicing and other processing on the first RAW image, convert the first RAW image into the above first YUV image, and convey the first YUV image to the IPE module, and the IPE module will perform noise reduction, cropping and other processing on the YUV image to obtain the final YUV image for display. On the other hand, the first IFE module will output the RAW image to the cache of the electronic device.

[0137] S112. The image acquisition module sends the second RAW image to the second IFE module.

[0138] S113. The second IFE module outputs the second RAW image.

[0139] S114. The second IFE module converts the second RAW image into the second YUV image.

[0140] S115. The second IFE module does not send the second YUV image to the IPE module.

[0141] It is understandable that steps S112 to S115 can be performed by the aforementioned second camera (auxiliary camera, including the aforementioned second acquisition module, second IFE module and IPE module). The second RAW image is generated by the second acquisition module included in the above-mentioned image acquisition module and sent to the IPE module.

[0142] It should be noted that the first RAW image and the second RAW image may be RAW images generated by the first acquisition module and the second acquisition module at the same time.

[0143] Specifically, the second acquisition module may also acquire optical signals at a certain acquisition rate, and convert the acquired optical signals into the second RAW image, and send the second RAW image to the second IFE module. In combination with the above description, it can be seen that in order to reduce power consumption, the electronic device may not use a binocular blur algorithm in the real-time preview stage, but may use a monocular blur algorithm to blur the background of the preview image, and the blurring of the image background does not need to be completed with the help of the YUV image output by the second camera. Therefore, after the second IFE module receives the second RAW image, the second IFE module may not transmit the second YUV image to the IPE module after converting the second RAW image into the second YUV image. However, in order to ensure the blurring effect of the photos taken in the ZSL shooting mode, the electronic device needs to use a binocular blurring algorithm to process the image to obtain the final photo for saving. After the user subsequently clicks the shooting control, the electronic device needs to simultaneously obtain the RAW images historically generated by the first camera and the second camera. Therefore, after the second IFE module receives the second RAW image, the second IFE module can convert the second RAW image into the above-mentioned second YUV image, and can choose to output the above-mentioned second RAW image to the cache for use by the electronic device in subsequent shooting.

[0144] It can be understood that in the ZSL mode, the RAW image generated by the second camera is only used as a reference frame when the image is subsequently taken, while the RAW image generated by the first camera needs to be used as a key frame when the image is subsequently taken. Therefore, in an embodiment of the present application, each time the first IFE module receives the RAW image transmitted by the above-mentioned first acquisition module, it is necessary to convert it into a YUV image and transmit the YUV image to the IPE module, and each frame of the RAW image needs to be output to the cache. However, each time the above-mentioned second IFE module receives the RAW image transmitted by the above-mentioned second acquisition module, in addition to not transmitting the YUV image obtained by converting the RAW image to the IPE module, the second IFE module can also output one frame of RAW image to the cache of the electronic device every other frame of RAW image, that is, the second IFE module only outputs one frame of RAW image to the cache after receiving two frames of RAW image. For details, please refer to the subsequent relevant description of steps S118-step S125.

[0145] In addition, it should be noted that, since steps S108 to S111 and steps S112 to S115 are respectively executed by two different cameras, the electronic device can synchronously execute steps S112 to S115 when executing steps S108 to S111.

[0146] S116. The IPE module sends a first preview image obtained after processing the first YUV image to the preview display module.

[0147] S117: The preview display module displays a first preview image.

[0148] The first preview image is an image obtained by processing the first YUV image using a monocular blur algorithm by the electronic device. Figure 5 The screen displayed in the preview box (B) of

[0149] S118. The image acquisition module sends the third RAW image to the first IFE module.

[0150] S119. The first IFE module outputs a third RAW image.

[0151] S120. The first IFE module converts the third RAW image into a third YUV image.

[0152] S121. The first IFE module sends the third YUV image to the IPE module.

[0153] Similarly, steps S118 to S121 can be performed by the aforementioned first camera. The third RAW image is generated by the first acquisition module included in the aforementioned image acquisition module and sent to the IPE module, and the third RAW image can be the latest RAW image generated by the aforementioned first acquisition module after generating the aforementioned first RAW image.

[0154] Specifically, after generating the first RAW, the first acquisition module will continue to acquire light signals in the scene and convert the light signals into the third RAW image. Therefore, the first acquisition module will continue to send the third RAW image to the first PIE module. In combination with the above description, it can be seen that after receiving the third RAW image, the first IFE module will process the third RAW image in two aspects, that is, the first IFE module will convert the third RAW image into the third YUV image, and transmit the third YUV image to the IPE module. In addition, the IPE module will output the RAW image to the cache of the electronic device.

[0155] S122. The image acquisition module sends the fourth RAW image to the second IFE module.

[0156] S123: The second IFE module does not output the fourth RAW image.

[0157] S124. The second IFE module converts the fourth RAW image into a fourth YUV image.

[0158] S125. The second IFE module does not send the fourth YUV image to the IPE module.

[0159] Steps S122 to S125 may be performed by the aforementioned second camera, wherein the second RAW image is generated by the second acquisition module included in the aforementioned image acquisition module and sent to the IPE module.

[0160] Similarly, the fourth RAW image and the third RAW image may be RAW images generated by the first acquisition module and the second acquisition module at the same time.

[0161] The second acquisition module converts the collected optical signal into the fourth RAW image, and sends the second RAW image to the second IFE module. After that, after the second IFE module converts the fourth RAW image into the second YUV image, it also does not transmit the second YUV image to the IPE module. However, unlike the aforementioned processing method for the second RAW image, the second IFE module may not output the fourth RAW image, that is, the second IFE module may not save the fourth RAW image in the cache of the electronic device.

[0162] In an embodiment of the present application, whether the first IFE module and the second IFE module output the YUV image to the IPE module or the IPE module, and whether the first IFE module and the second IFE module output the RAW transmitted from the acquisition module to the cache, can be determined by the indication information returned by the IPE module. Taking the above-mentioned IPE module as an example, after the shooting control module and the preview display module are started, the preview display module in the application layer can send frame request instructions to the hardware abstraction layer in real time. Each frame request instruction represents that a frame of preview image needs to be displayed on the screen. Before the above-mentioned second acquisition module collects the light signal to generate the above-mentioned second RAW image, the preview display module can send the frame request instruction request1 (request1 is used to request the hardware abstraction layer to generate the above-mentioned second RAW image and send it for display after processing) to the hardware abstraction layer. The hardware abstraction layer can send the feature feature1 corresponding to request1 to the IPE module. The IPE module can determine an indication information based on the information carried by feature1. The indication information can be used to transmit back to the second IFE module and instruct the second IFE module to open the RAW image output port (that is, the channel for transmitting the RAW image to the cache). Therefore, when the second IFE module receives the above-mentioned second RAW image sent by the second acquisition module, the second IFE module can output the above-mentioned second RAW image to the cache of the electronic device. Similarly, before the second acquisition module collects the optical signal to generate the fourth RAW image, the preview display module can send the frame request instruction request2 to the hardware abstraction layer, and the hardware abstraction layer can send the feature feature2 corresponding to request2 to the IPE module. The IPE module can also determine an indication message based on the information carried by feature2, and the indication message can be used to transmit it back to the second IFE module and instruct the second IFE module not to open the RAW image output port. Therefore, when the second IFE module receives the fourth RAW image sent by the second acquisition module, the second IFE module may not output the fourth RAW image to the cache of the electronic device.

[0163] S126. Send a second preview image obtained after processing the fourth YUV image.

[0164] S127: Display the second preview image.

[0165] Similarly, the second preview image is also an image obtained by the electronic device processing the fourth YUV image using a monocular blur algorithm. Figure 5 The screen displayed in the preview box (B) of

[0166] Afterwards, the first camera and the second camera in the electronic device can continue to output images according to the output logic shown in the above steps. Specifically, the first acquisition module in the first camera transmits the RAW image to the above-mentioned first IFE module, and the first IFE module converts each received frame of the RAW image into a YUV image and sends it to the IPE module, and the first IFE module outputs each received frame of the RAW image to the cache of the electronic device; the IPE module crops the received YUV image, performs monocular blur processing, and then outputs it to the screen of the electronic device for user preview. At the same time, the second acquisition module in the second camera transmits the RAW image to the above-mentioned second IFE module. After the second IFE module converts each received frame of the RAW image into a YUV image, it does not send it to the IPE module (or the second IFE module may not convert the RAW image into a YUV image). Correspondingly, the IPE module will not output the YUV image when it cannot receive the YUV image transmitted by the IPE module; however, the second IFE module can output one frame of RAW image to the cache of the electronic device every other frame. For example, after receiving the above-mentioned fourth cache image, the second IFE module continuously receives multiple frames of RAW images. The second IFE module can output the first frame of RAW image to the cache of the electronic device without outputting the second frame of RAW image, and output the third frame of RAW image to the cache of the electronic device without outputting the fourth frame of RAW image, and so on.

[0167] If the user subsequently clicks the capture control, for example Figure 5 In the example shown in (B) of FIG. 1 , the user touches the shooting control 413, and the electronic device selects a number of RAW images from the RAW images output to the cache by the first IFE module, and selects the same number of RAW images corresponding to the aforementioned several frames of images from the RAW images output to the cache by the second IFE module, and performs image background blurring, multi-frame image fusion and other processing on the obtained RAW images, and then uses the last obtained frame of image as the photographed photo (for details, please refer to the subsequent Figure 6-Figure 8 The relevant instructions are not repeated here) and stored in the electronic device.

[0168] by Figure 6 Take this as an example to illustrate. Figure 5 In response to the user's touch operation on the shooting control 413 shown in (B) in FIG. 1 , the electronic device may display the following Figure 6 The user interface 51 shown in (A) in FIG. Figure 6 As shown in (A), the user interface 51 is a shooting interface of the portrait mode in the camera application, and the user interface 51 may include a viewfinder 511, a background blur control 512, and a control for displaying the image that has been shot 513.

[0169] The viewfinder 511 will continue to display the preview image to the user in real time. A face image 5111 and a tree image 5112 are displayed in the viewfinder 511. When the background is blurred, the electronic device can determine the face image 5111 as the foreground (subject) and the tree image 5112 as the background. Since the background blur control 512 is already on at this time, the tree image 5112 has been blurred. However, since the preview image displayed by the electronic device uses a monocular blur algorithm to blur the image, the blur effect of the preview image may be relatively rough, and the background may be blurred or the subject may be blurred. Figure 6 As shown in (A), the face image 5111 is the foreground, and the hair part 5111A is also the foreground, but the hair part 5111A is still blurred.

[0170] The captured image echo control 513 may display a thumbnail of the photo obtained after the user clicks the capture control.

[0171] In response to Figure 6 The electronic device may display the following information when the user touches the captured image display control 513 as shown in (A) of FIG. Figure 6 The user interface 61 shown in (B) in FIG. The user interface 61 may be an application interface of a gallery application, which may include an image 611 for displaying photos taken by the user in history, that is, a photo obtained by processing the electronic device based on a plurality of RAW images output by the first IFE module and the same number of RAW images output by the second IFE module. In the image 611, the face image 6111 is determined as the foreground (subject), and the tree image 6112 is determined as the background. In combination with the signed description, it can be seen that the image 611 is blurred using a binocular blur algorithm, so the blur effect of the preview image is more delicate than that of the preview image, wherein the face image 6111 has a clear outline, and there is no part that is blurred incorrectly, and the tree image 6112 as the background image is also accurately blurred.

[0172] Figure 7 The specific image output method when the electronic device implements the shooting method provided in this application in the ZSL mode is shown.

[0173] like Figure 7 (A) in the figure shows the image output method of the electronic device in the preview stage.

[0174] During the preview stage, the first camera outputs a YUV image for display (such as the first YUV image and the third YUV image mentioned above) at a first frame rate. These YUV images will eventually be drawn on the screen by the electronic device for user preview and display after the electronic device has processed the background blur based on a monocular blur algorithm. At the same time, the first camera will also output a RAW image (such as the first RAW image mentioned above) for conversion into the aforementioned YUV image to the cache of the electronic device at the first frame rate. In other words, each time the first camera generates a RAW image, the first camera will output the RAW image to the cache of the electronic device. In addition, the first camera will convert the RAW image into a YUV image and then transmit it to the screen of the electronic device (for example, first transmit it to the CPU or GPU, and the CPU or GPU will draw it to the screen of the electronic device) for user preview. Figure 7 Taking image 701 and image 702 in the figure as an example (image 701 and image 702 may be the first YUV image and the first RAW image in the aforementioned description respectively), image 702 may be a RAW image generated by the first acquisition module in the first camera, and image 701 may be a YUV image obtained after image 702 is processed by the first IFE module and the IPE module in the first camera.

[0175] In the preview stage, since the YUV image input port of the IPE module (i.e., the data interface of the IPE module for receiving the YUV image transmitted by the second IFE module) is always in a closed state under the control of the electronic device, the second camera does not output the YUV image for display throughout the whole process. However, under the control of the electronic device, when the first camera generates a frame of RAW image, the second camera will also generate a frame of RAW image, but the metadata output port of the first IFE module in the second camera (i.e., the data interface of the second IFE module for outputting the RAW image) will be opened every other frame with the reception of the RAW image, so the second camera will choose to output a frame of RAW image to the cache of the electronic device every two frames of RAW image generated. For example, the second camera can output image 704 to the cache of the electronic device after generating image 704, but does not output image 704 after generating image 705 (at this time, the first camera still outputs image 703 to the cache of the electronic device, and image 703 and image 704 are two frames of RAW images generated by the first camera and the second camera at the same time). Therefore, in fact, the frame rate of the RAW image output by the second camera is half of the frame rate of the RAW image output by the first camera.

[0176] like Figure 7 (B) in the figure shows the way in which the electronic device outputs images during the shooting stage.

[0177] During the shooting stage, the first camera and the second camera can continue to output images in the same manner as in the preview stage. However, in order to obtain the photographed photos, the electronic device can select several frames from the RAW images output by the first camera to the cache and the RAW images output by the second camera to the cache for processing to obtain the above-mentioned photos. Specifically, the electronic device can determine image 706 and image 707 from the RAW images output by the first camera to the cache, and based on the frame numbers of the two images (the number can reflect the generation time of the RAW images, and the two RAW images with the same frame number have the same generation time), determine image 708 and image 709 from the RAW images output by the second camera to the cache, wherein image 708 has the same frame number as image 706, and image 709 has the same frame number as image 707. Afterwards, the electronic device uses a binocular blur algorithm to process image 708 and image 706 to obtain image 710 with a blur effect. Correspondingly, the electronic device can use a binocular blur algorithm to process image 707 and image 709 to obtain image 711 with a blur effect, and fuse the information in image 710 and image 711 through a multi-frame fusion algorithm (for example, using image 710 as a key frame and image 711 as a reference frame, and superimposing the information in image 711 onto image 710), and store the final image 712 as the above-mentioned photo in the electronic device (for example, in a gallery).

[0178] Of course, in some embodiments, the electronic device may also select a larger number of images (for example, three frames of images each) from the RAW images output from the first camera to the cache and the RAW images output from the second camera to the cache, and obtain the above-mentioned photos according to the above-mentioned processing method. This application is not limited to this.

[0179] In addition, in some embodiments, when the electronic device uses the binocular blur algorithm and the multi-frame fusion algorithm to obtain the above-mentioned first preview image and the second preview image, it can also first determine the key frame and the reference frame in the RAW image output by the first camera, and after fusing the RAW image output by the first camera, obtain a fused frame of image, and then calculate the depth of field information of the image based on the reference frame and the RAW image corresponding to the reference frame (that is, the RAW image output by the second camera and generated at the same time as the reference frame), and blur the background of the above-mentioned fused frame of image based on the depth of field information, and store the blurred image as the above-mentioned photo in the electronic device (for details, please refer to the aforementioned Figure 1 The relevant instructions will not be repeated here).

[0180] Optionally, after the user clicks the shooting control, the electronic device may use other methods to process the RAW images in the historical cache to obtain the above-mentioned photos, which may be determined by the algorithm logic of the multi-frame fusion algorithm and the binocular blur algorithm used by the electronic device, and this application does not limit this. However, in order to ensure the background blur effect of the photo and the absence of delay between the photo screen and the user's shooting time, the historical cached RAW image obtained by the electronic device (that is, the RAW image stored before the user clicks the shooting control) must contain both the RAW image of the first camera's historical output and the RAW image of the second camera's historical output.

[0181] In addition, in combination with the above description, it can be known that the RAW images output by the above-mentioned first IFE module and the above-mentioned second IFE module will be saved in the cache (hereinafter, the RAW images output by the first IFE module and saved in the cache are referred to as the RAW images output by the main camera, and the RAW images output by the second IFE module and saved in the cache are referred to as the RAW images output by the auxiliary camera). In addition, in order to be able to determine a pair of RAW images with the same generation time from the RAW images output by the two cameras in the subsequent shooting and frame selection process, when outputting the RAW images to the cache, the electronic device will also add frame numbers to the RAW images, wherein the RAW images generated at the same time (for example, the above-mentioned first RAW image and the second RAW image) have the same frame numbers corresponding to them. When selecting the RAW images, the electronic device can determine the required RAW images and the frame numbers corresponding to these RAW images from the RAW images output by one of the cameras, and then select the RAW images with the same frame numbers from the RAW images output by the other camera based on the frame numbers corresponding to these RAW images.

[0182] However, due to storage space limitations and the fact that the second IFE stores RAW images in interval frames, when the user subsequently clicks on the shooting control and the electronic device selects corresponding RAW images from the RAW images output by the main camera and the RAW images output by the auxiliary camera, after the electronic device determines multiple frames of RAW images from the RAW images output by the main camera, the electronic device may not be able to completely find RAW images with the same frame numbers of the multiple frames of RAW images in the RAW images output by the other camera.

[0183] by Figure 8 For the purpose of explanation, it is assumed that the electronic device stores 50 frames of RAW images generated by the first camera and the second camera respectively, and the frame numbers are 1-50, and the first cache queue for storing the RAW images output by the first camera can store up to 10 frames of RAW images, and the second cache queue for storing the RAW images output by the second camera can also store up to 10 frames of RAW images. When a new RAW image needs to be cached in the queue, the earliest cached RAW image in the queue will be cleared. Figure 8As shown in (A), the first cache queue caches the 10 RAW images most recently output by the first camera, and the frame numbers corresponding to the 10 RAW images cached in the first cache queue are 41-50. The second cache queue also caches the 10 RAW images most recently output by the second camera. However, since the second camera outputs RAW images at intervals of one frame, the frame numbers corresponding to the 10 RAW images cached in the second cache queue are 32, 34, 36, 38, 40, 42, 44, 46, 48, and 50, respectively. After the user clicks on the shooting control, assuming that the electronic device needs to select 3 frames of images from the first cache queue and the second cache queue respectively, and obtain the above-mentioned photos according to the processing logic in the above description, and the electronic device selects three RAW images with frame numbers 46, 47, and 48 from the first cache queue, then the electronic device needs to select three RAW images with the same frame numbers from the second cache queue based on the frame numbers of these three RAW images. However, from Figure 8 It can be seen that there are only RAW images with frame numbers 46 and 48 in the second cache pair column, but there is actually no RAW image with frame number 47. Therefore, the electronic device may not be able to output the above photos normally in the end.

[0184] In view of the above defects, in a possible implementation, the electronic device can reduce the length of the second cache queue by half and only retain the latest five RAW images to ensure that any RAW image in the second cache queue has a RAW image with the same frame number as the RAW image in the first cache queue. Then, the electronic device can first determine the frame numbers of the three required RAW images from the second cache queue, and then find another three RAW images corresponding to the frame numbers of the three RAW images from the first cache queue.

[0185] like Figure 8As shown in (B), after the length of the cache in the second cache queue is halved, the frame numbers corresponding to the 10 most recently cached RAW images in the first cache queue are still 41-50, and the frame numbers corresponding to the 5 most recently cached RAW images in the second cache queue are 42, 44, 46, 48, and 50. It can be understood that the set {42, 44, 46, 48, 50} is included in the set {41, 42, 43, 44, 45, 46, 47, 48, 49, 50}, so after the user clicks the shooting control, no matter what the frame number of the required RAW image is determined by the electronic device from the second cache queue, the electronic device can select the RAW image with the same frame number from the first cache queue based on these frame numbers. For example, assuming that the electronic device determines from the second cache queue that the three required RAW images have frame numbers 44, 46, and 48, respectively, the electronic device can select three RAW images with frame numbers 44, 46, and 48 from the first cache queue, and obtain the above-mentioned photo through these 6 RAW images.

[0186] It is understandable that the embodiment of the present application is introduced by taking image capture in portrait mode as an example, but the embodiment of the present application is not limited to portrait mode, and the above-mentioned shooting method can also be used in other shooting modes involving multi-camera outflow, and the embodiment of the present application is not limited to this. In addition, in some embodiments, when the second camera outputs a RAW image, it can also output a frame of RAW image to the cache of the electronic device every two frames (that is, output a frame of RAW image for every three frames of RAW image generated), as long as the frame rate of the second IFE module in the second camera outputting the RAW image to the cache is smaller than the frame rate of the second IFE module receiving the RAW image, and the present application is not limited to this.

[0187] Fig. 9 The process of the electronic device executing the above-mentioned shooting method is shown from the perspective of the Android system architecture. Fig. 9 As shown, the architecture involved in the process of the electronic device executing the aforementioned shooting method may include the following three layers: application layer, application framework layer, and hardware abstraction layer. The specific functions of each layer of architecture in the Android system can refer to the aforementioned related instructions and will not be repeated here. In the process of the electronic device executing the aforementioned shooting method, the application layer, application framework layer, and hardware abstraction layer can work together to complete the entire preview process, specifically:

[0188] The application layer can respond to user operations, such as the user opening the shooting application, and continuously send preview frame request instructions to the application framework layer, and each frame request instruction corresponds to a frame of preview image displayed on the electronic device screen. Alternatively, the application layer can respond to user operations, such as the user clicking the shooting control, and continuously send shooting and image request instructions to the application framework layer, and the shooting and image request instructions can be used to instruct the hardware abstraction layer to use the cached RAW image to obtain a photo.

[0189] After that, the application framework layer will further send the preview frame request instruction or the capture image request instruction transmitted by the application layer to the hardware abstraction layer, and the hardware module in the hardware abstraction layer will output the corresponding RAW image and / or YUV image according to the instruction. For details, please refer to the above Figure 2 and Figure 3 The relevant instructions will not be repeated here.

[0190] Afterwards, the hardware layer can pass the generated preview image (the preview image can be some drawing instructions at this time) back to the application layer through the application framework layer, and SurfaceFlinger draws the preview image on the screen of the electronic device for the user to browse according to the drawing instructions.

[0191] Next, the electronic device provided by the embodiments of the present application is introduced.

[0192] The electronic device may be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) or a special camera (such as a SLR camera, a card camera), etc. The present application does not impose any restrictions on the specific type of the electronic device. Specifically, the electronic device 100 may be the electronic device in the aforementioned description.

[0193] Fig.10 The structure of the electronic device is exemplarily shown.

[0194] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, a button 190, a motor 191, a camera 193, a display screen 194, etc.

[0195] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0196] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0197] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0198] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0199] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0200] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 may be coupled to a charger, a flash, a camera 193, etc. through different I2C bus interfaces.

[0201] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0202] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0203] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0204] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0205] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0206] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker, a receiver, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module or other functional modules.

[0207] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0208] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0209] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0210] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0211] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include N cameras 193, where N is a positive integer greater than 1.

[0212] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0213] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0214] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0215] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0216] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0217] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0218] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0219] In the present application, the camera 193 includes at least two cameras, including a first camera and a second camera, both of which are front cameras of the electronic device or rear cameras of the electronic device. Optionally, the first camera can be a wide-angle camera and the second camera can be an ultra-wide-angle camera; or the first camera can be a telephoto camera and the second camera can be a wide-angle camera.

[0220] In an embodiment of the present application, the processor 110 can control the frame output mode of the first camera and the second camera in the ZSL mode.

[0221] In the preview stage of the ZSL shooting mode, the first ISP (including the first IFE) in the first camera may convert the RAW image generated by the first sensor in the first camera into a YUV image and output it, and the second ISP (including the second IFE) in the second camera may not output the RAW image generated by the second sensor in the second camera. After processing the YUV image output by the first camera based on the monocular blur algorithm, the processor 110 may work with the GPU to draw the processed image to the display screen 194 for display.

[0222] In addition, in the preview stage of the ZSL shooting mode, the first camera outputs a RAW image at a first frame rate, and the second camera outputs a RAW image at a second frame rate, wherein the first frame rate is less than the second frame rate. The RAW images output by the first camera and the second camera can be cached in a cache area of ​​the electronic device, and the cache area can be a storage area in a memory included in the processor 110, a storage area in an external memory, or a storage area in the internal memory 121.

[0223] During the shooting stage in the ZSL shooting mode, in response to the user's touch operation on the shooting control, the processor 110 can select RAW images with the same number of frames from the RAW images output by the first camera and the RAW images output by the second camera, respectively, to obtain M groups of RAW images with the same generation time (i.e., two RAW images with the same frame number). The processor 110 can use the binocular blur algorithm to process each group of RAW images to obtain M frames of images (the backgrounds of these images have a blurred effect), and use a multi-frame fusion algorithm to fuse the M frames of images to obtain the final photos that need to be stored.

[0224] Optionally, after the user clicks the shooting control, the electronic device may use other methods to process the RAW images in the historical cache to obtain the above-mentioned photos, which may be determined by the algorithm logic of the multi-frame fusion algorithm and the binocular blur algorithm used by the electronic device, and this application does not limit this. However, in order to ensure the background blur effect of the photo and the absence of delay between the photo screen and the user's shooting time, the historical cached RAW image obtained by the electronic device (that is, the RAW image stored before the user clicks the shooting control) must contain both the RAW image of the first camera's historical output and the RAW image of the second camera's historical output.

[0225] An embodiment of the present application also provides an electronic device, which includes: one or more processors and a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method shown in the aforementioned embodiment.

[0226] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0227] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0228] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A shooting method, It is characterized in that The method comprises: Controlling the first camera to output a RAW image at a first frame rate; Controlling the second camera to output a RAW image at a second frame rate; The first frame rate is greater than the second frame rate.

2. The method according to claim 1, It is characterized in that The first frame rate is a frame rate at which the first camera and the second camera generate RAW images.

3. The method according to claim 1 or 2, It is characterized in that The first frame rate is twice the second frame rate, and controlling the first camera to output the RAW image at the first frame rate includes: Controlling the first camera to output each frame of RAW image generated by the first camera; The controlling the second camera to output a RAW image at a second frame rate comprises: The second camera is controlled to output one RAW image of the two RAW image frames when two RAW image frames are generated.

4. The method according to any one of claims 1 to 3, It is characterized in that The method further comprises: controlling the first camera to output a YUV image at the first frame rate, Processing the first YUV image output by the first camera based on a monocular blur algorithm to obtain a first preview image; The first preview image is displayed in a preview area of ​​a display screen.

5. The method according to any one of claims 1 to 4, It is characterized in that The method is applied to a shooting mode with a zero-second delay, and the method further comprises: In response to a user's operation on a shooting control, a first photo is saved, where the first photo is obtained by processing at least one frame of a first RAW image output by the first camera and at least one frame of a second RAW image output by the second camera based on a binocular blur algorithm.

6. The method according to any one of claims 1 to 5, It is characterized in that The first frame rate is n times the second frame rate, the RAW images output by the first camera are cached in a first cache queue, and the RAW images output by the second camera are cached in a second cache queue. When the first cache queue and the second cache queue are full, the RAW images stored in the second cache queue are n times the number of RAW images in the first cache queue, and n is a positive number.

7. The method according to claim 5 or 6, It is characterized in that Before saving the first photo, the method further includes: Determine at least one second RAW image from the RAW image output by the second camera; Determine at least one first RAW image frame from the RAW images output by the first camera based on the frame numbers of the at least two RAW images, the number of the at least one first RAW image frame is the same as the number of the at least one second RAW image frame; The at least one frame of the first RAW image is fused based on a multi-frame fusion algorithm to obtain a first image; Processing the first key frame and the second key frame based on a binocular blur algorithm to obtain depth of field information of the first key frame, wherein the first key frame is a RAW image with better image quality in the at least one first RAW image, and the second key frame is a RAW image in the at least one second RAW image with the same frame number as the first key frame; The background of the first image is blurred based on the depth of field information to obtain the first photo.

8. The method according to any one of claims 1 to 7, It is characterized in that The first camera may be a wide-angle camera, and the second camera may be an ultra-wide-angle camera; Alternatively, the first camera may be a telephoto camera, and the second camera may be a wide-angle camera.

9. An electronic device, It is characterized in that The electronic device comprises: one or more processors, a memory and a display screen; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 8.

10. A chip system, It is characterized in that The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-8.

11. A computer-readable storage medium comprising instructions, It is characterized in that When the instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 8.

Images