Photographing method and device, electronic equipment and storage medium

Abstract

The invention relates to a photographing method and device, electronic equipment and a storage medium. The photographing method comprises the steps that in response to the fact that a camera application is in a photographing mode, a first camera and a second camera are controlled to acquire original images respectively to obtain a first image and a second image, and the original images are images in a RAW data format; performing dynamic range adjustment on the first image to obtain a high dynamic range (HDR) image of the first image; and performing image fusion based on the HDR image, the first image and the second image to obtain a photographed image. Through the embodiment of the invention, the image processing time in the photographing process can be reduced, and the photographing time is further reduced.

Description

Technical Field

[0001] This disclosure relates to the field of photography technology, and in particular to photography methods, apparatus, electronic devices and storage media. Background Technology

[0002] With the development of terminal technology, multi-camera-assisted photography is being used more and more widely in user scenarios. For example, portrait mode photography or ultra-wide-angle mode photography.

[0003] Taking portrait mode as an example, current portrait mode produces images with a depth-of-field effect, meaning that areas outside the portrait are blurred. To enable photos to have a corresponding dynamic range effect in high dynamic range scenes, a High Dynamic Range (HDR) algorithm needs to be overlaid on the portrait mode.

[0004] The portrait algorithms in related technologies require the input image to be in YUV format. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this disclosure provides a photographing method, apparatus, electronic device, and storage medium.

[0006] According to a first aspect of the present disclosure, a method for taking a picture is provided, comprising: responding to a camera application being in a picture mode, controlling a first camera and a second camera to acquire original images respectively to obtain a first image and a second image, wherein the original image is an image in RAW data format; performing dynamic range adjustment on the first image to obtain a high dynamic range (HDR) image of the first image; and performing image fusion based on the HDR image, the first image, and the second image to obtain a picture image.

[0007] In one embodiment, the step of image fusion based on the HDR image, the first image, and the second image to obtain a captured image includes: determining the disparity value of the first image and the second image, and determining the depth map of the first image based on the disparity value, wherein the disparity value characterizes the degree of positional difference between matching pixel pairs in the first image and the second image; using the first image as a reference frame, and performing image fusion on the HDR image and the depth map to obtain the captured image.

[0008] In another embodiment, the step of image fusion of the HDR image and the depth map includes: determining a brightening template corresponding to the first image based on the HDR image and the first image, wherein the brightening template is used to adjust the dynamic range corresponding to the first image; and performing image fusion of the brightening template and the depth map.

[0009] In another embodiment, determining the depth map of the first image based on the disparity value includes: establishing a target mapping relationship, wherein the target mapping relationship is a mapping relationship between the disparity value and depth information; determining the depth information corresponding to the pixels in the first image in the pixel pair based on the pixel pair, the disparity value corresponding to the pixel pair, and the target mapping relationship; and determining the depth map corresponding to the first image based on the depth information.

[0010] In another embodiment, the photographed image is obtained by calling a preloaded image processing program to process the first image and the second image to obtain the photographed image. The image processing program is used to process the original image to obtain the photographed image.

[0011] According to a second aspect of the present disclosure, a photographing device is provided, comprising: a control unit, configured to control a first camera and a second camera to acquire original images respectively, thereby obtaining a first image and a second image, wherein the original image is an image in RAW data format; and a processing unit, configured to perform dynamic range adjustment on the first image to obtain a high dynamic range (HDR) image of the first image, and to perform image fusion based on the HDR image, the first image, and the second image to obtain a photographed image.

[0012] In one embodiment, the processing unit performs image fusion based on the HDR image, the first image, and the second image to obtain a captured image in the following manner: determining the disparity values ​​of the first image and the second image, and determining the depth map of the first image based on the disparity values, wherein the disparity values ​​characterize the degree of positional difference between matching pixel pairs in the first image and the second image; using the first image as a reference frame, and performing image fusion on the HDR image and the depth map to obtain the captured image.

[0013] In another embodiment, the processing unit performs image fusion of the HDR image and the depth map in the following manner: based on the HDR image and the first image, a brightening template corresponding to the first image is determined, the brightening template being used to adjust the dynamic range corresponding to the first image; and the brightening template and the depth map are then fused.

[0014] In another embodiment, the processing unit determines the depth map of the first image based on the disparity value in the following manner: establishing a target mapping relationship, wherein the target mapping relationship is a mapping relationship between the disparity value and the depth information; determining the depth information corresponding to the pixels in the first image in the pixel pair based on the pixel pair, the disparity value corresponding to the pixel pair, and the target mapping relationship; and determining the depth map corresponding to the first image based on the depth information.

[0015] In another embodiment, the processing unit obtains the photographed image by calling a preloaded image processing program to process the first image and the second image to obtain the photographed image. The image processing program is used to process the original image to obtain the photographed image.

[0016] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the photographing method described in the first aspect or any embodiment of the first aspect.

[0017] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor, enable the processor to perform the photographing method described in the first aspect or any embodiment of the first aspect.

[0018] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: by controlling the first camera and the second camera to acquire the original images respectively, since the images acquired by the different cameras are in RAW format, the processing steps of format conversion due to the different image formats acquired by the different cameras can be avoided in the process of image processing based on the first image and the second image to obtain the captured image, thereby reducing the image processing time during the capture process and thus reducing the capture time.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0021] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0022] Figure 2A This is a flowchart illustrating a photographing method according to an exemplary embodiment.

[0023] Figure 2B This is a flowchart illustrating a photographing method according to an exemplary embodiment.

[0024] Figure 3A This is a schematic diagram illustrating the activation of an image processing program according to an exemplary embodiment.

[0025] Figure 3B This is a flowchart illustrating a photographing method according to an exemplary embodiment.

[0026] Figure 3C This is a schematic diagram illustrating an image processing program preloading according to an exemplary embodiment.

[0027] Figure 4 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0028] Figure 5 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0029] Figure 6A This is a flowchart illustrating a method for determining the depth map of a first image according to an exemplary embodiment.

[0030] Figure 6B This is a schematic diagram illustrating an image post-processing stage according to an exemplary embodiment.

[0031] Figure 7 This is a block diagram of a photographing device according to an exemplary embodiment.

[0032] Figure 8 This is a block diagram illustrating an apparatus for taking a photograph according to an exemplary embodiment.

[0033] Figure 9 This is a block diagram illustrating an apparatus for taking a photograph according to an exemplary embodiment. Detailed Implementation

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0035] With the development of terminal technology, multi-camera-assisted photography is being used more and more widely in user scenarios. For example, portrait mode photography or ultra-wide-angle mode photography.

[0036] Taking portrait mode as an example, current portrait mode produces images with a depth-of-field effect, meaning that areas outside the subject are blurred. To enable photos to have a corresponding dynamic range in high dynamic range scenes, a High Dynamic Range (HDR) algorithm needs to be overlaid on the portrait mode. For ease of understanding, this will be explained through... Figure 1 This paper explains the camera processing flow in the portrait photography mode of related technologies. Figure 1 This is a schematic flowchart illustrating an image processing method according to an exemplary embodiment. Figure 1 In this process, the terminal ultimately acquires the captured image, which can be divided into two stages: the image acquisition stage and the post-processing stage. In the image acquisition stage, in response to the terminal being in photo mode, the terminal controls the main camera and the secondary camera to capture images respectively. The image captured by the main camera is in raw image format (hereinafter referred to as RAW image), and the image captured by the secondary camera is in Luminance-Chrominance (YUV) encoded image format (hereinafter referred to as YUV image). After the image acquisition stage is completed, the images captured by the main camera and the secondary camera are sent to the processing software corresponding to the post-processing stage. This allows the processing software to acquire N frames of RAW images captured by the main camera and N frames of YUV images captured by the secondary camera for image processing, where N is an integer greater than or equal to 1. For example, in the post-processing stage, corresponding image processing operations are performed on the RAW images captured by the main camera and the YUV images captured by the secondary camera. For the RAW image captured by the main camera, image frame selection is performed using virtual allocation nodes. The selected images are then format-converted, transforming the RAW image from the main camera into a YUV image corresponding to the main camera. This YUV image is then processed using a High Dynamic Range (HDR) algorithm to obtain a YUV format HDR image. Furthermore, the YUV image from the main camera and the YUV image from the secondary camera are combined using a depth map calculation algorithm to obtain a YUV format depth map. Finally, a portrait mode image is generated by fusing the YUV format HDR image and the YUV format depth map using a portrait algorithm.

[0037] And through Figure 1 As described in the relevant examples, the portrait algorithms in related technologies require YUV format images as input. Therefore, after the main camera acquires the RAW format image, a format conversion operation is required. This significantly impacts the time required to output the final image.

[0038] Based on this, this disclosure proposes a method for taking pictures, applied to a terminal. By controlling different cameras to take pictures of the same subject, images are obtained from each camera, and the acquired images have the same file format. Then, image processing is performed on the original images acquired by each camera to obtain the final captured image.

[0039] It should be noted that, in some embodiments, the terminal includes, but is not limited to, at least one of the following: mobile phone, wearable device, Internet of Things device, car with communication function, smart car, tablet computer, computer with wireless transceiver function, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal device in industrial control, wireless terminal device in self-driving, wireless terminal device in remote medical surgery, wireless terminal device in smart grid, wireless terminal device in transportation safety, wireless terminal device in smart city, and wireless terminal device in smart home.

[0040] For ease of understanding, the embodiments disclosed herein will be described using a mobile phone type terminal as an example. In this embodiment, when the terminal's camera is in photo mode, it utilizes at least two cameras to capture images. For ease of understanding, the following embodiments will use the example of a terminal utilizing two cameras for image capture as an example for further illustrative description.

[0041] It should be understood that the photographing method proposed in the embodiments of this disclosure is also applicable to other applicable scenarios.

[0042] Figure 2A This is a flowchart illustrating a photographing method according to an exemplary embodiment, such as... Figure 2A As shown, the photo-taking method used in the terminal includes the following steps.

[0043] In step S11, in response to the camera application being in shooting mode, the first camera and the second camera are controlled to acquire the original image respectively, resulting in the first image and the second image, wherein the original image is an image in RAW data format.

[0044] In step S12, the dynamic range of the first image is adjusted to obtain an HDR image of the first image.

[0045] In step S13, image fusion is performed based on the HDR image, the first image, and the second image to obtain the captured image.

[0046] In this embodiment of the disclosure, the first camera and the second camera are controlled to acquire the original images respectively. Since the images acquired by the different cameras are in RAW format, the processing steps of format conversion due to the different image formats acquired by the different cameras can be avoided in the process of image processing based on the first image and the second image to obtain the captured image. This solves the image processing time problem in the process of taking pictures and reduces the shooting time.

[0047] It's understandable that by controlling the first and second cameras to acquire the original images, the original images have higher image quality than non-original image formats, and the noise reduction effect on the original images is also better during image processing. Therefore, using the original images for image processing can result in a better quality final photographed image.

[0048] In some embodiments, the dynamic range adjustment of the first image can be achieved in the following manner:

[0049] (1) Preprocess the first image (e.g., denoising, white balance correction, etc.).

[0050] (2) Call the corresponding feature matching algorithm to find the corresponding feature points in multiple images and perform matching. For example, find some corner points in the left image, and then find the points in the right image that match the corner points in the left image.

[0051] (3) For the matched feature point pairs, calculate the difference in their horizontal positions in the left and right images.

[0052] (4) Generate a depth map from the disparity map. The depth map represents the distance between each pixel and the camera. For example, if the disparity value is 5 pixels, the depth value of that pixel in the depth map is 5 meters.

[0053] (5) Map the wide dynamic range of the HDR image to the limited dynamic range of the standard display device, while preserving the naturalness and detail of the image as much as possible, to generate a low dynamic range (LDR) image.

[0054] (6) Combine the tone-mapped LDR image and the depth map to obtain the final HDR image. For example, if a pixel has a value of 255 in the LDR image and a value of 5 meters in the depth map, then the pixel in the HDR image will have a value of 255.

[0055] It is clear that the above-mentioned HDR processing of the first image to obtain an HDR image involves operations such as feature point matching, and this process needs to be repeated for each portrait processing step, resulting in a large computational load. In some embodiments, HDR processing of the first image during portrait processing can be performed by outputting a brightening template to the corresponding image processing program, thereby reducing the processing time.

[0056] It should be noted that the terms "image in original image format," "original image," "original image," and "RAW image" are interchangeable. For ease of understanding, the term "RAW image" will be used to describe images in the original image format in the following examples.

[0057] For ease of understanding, the embodiments disclosed herein will be explained through... Figure 2B As Figure 1 Comparative examples of related embodiments are used to illustrate Figure 2A The described related photography methods are illustrated by example.

[0058] Figure 2B This is a flowchart illustrating a photographing method according to an exemplary embodiment. Figure 2B As shown, the system controls the main camera to acquire RAW images and the secondary camera to acquire RAW images. For example, the main camera acquires N frames of RAW images of the target subject, and the secondary camera acquires N frames of RAW images of the target subject. Then, through a corresponding image acquisition process (e.g., using a set frame selection algorithm to determine RAW1 from the N frames acquired by the main camera, and / or RAW2 from the N frames acquired by the secondary camera), the determined RAW1 and RAW2 images are sent to the corresponding modules in the post-processing stage for appropriate post-processing operations.

[0059] Understandable Figure 2B Related embodiments relative to Figure 1 In this embodiment, both the main camera and the secondary camera acquire RAW images, and the acquired RAW images are sent to the post-processing stage. Therefore, compared to... Figure 1 In this embodiment, the process of converting image file formats can be reduced in the post-processing stage, thereby saving shooting time.

[0060] Based on this, the above embodiments can shorten the image capture time during the image acquisition stage. Similarly, adjustments can be made to the post-processing stage to further shorten the image capture time.

[0061] Understandably, post-processing typically involves calling different image processing programs to perform corresponding image processing operations on the acquired images. The activation of an image processing program requires a startup and loading process. In related technologies, the activation process of an image processing program is usually as follows: Figure 3A As shown. Figure 3A This is a schematic diagram illustrating the activation of an image processing program according to an exemplary embodiment. Figure 3A It is known that in related technologies, image processing programs are typically invoked during the post-processing stage after the image acquisition stage (e.g., a first image processing program, and / or a second image processing program). Understandably, this invocation requires a program pre-loading process. Consequently, the image processing program cannot process the image immediately upon acquisition (e.g., it needs to wait for the image processing program to start after acquisition), thus increasing the capture time.

[0062] Based on this, the present disclosure proposes a photographing method to solve the above-mentioned technical problems. The method is as follows: Figure 3B As shown, Figure 3B This is a flowchart illustrating a photographing method according to an exemplary embodiment, the method comprising the following steps.

[0063] In step S21, the image processing program is preloaded.

[0064] In step S22, a preloaded image processing program is invoked to process the first image and the second image to obtain a photographed image.

[0065] The image processing program is used to process the original image to obtain the photographed image.

[0066] In this embodiment, the image processing program is preloaded and then invoked to process the first and second images. Since the image processing program is preloaded before processing the first and second images, the loading time of the image processing program upon acquiring the first and second images is reduced, thereby reducing the time required to take the picture.

[0067] It is understandable that the first and second images can be interpreted as images to be loaded. The preloaded image processing program can be understood as an image processing program that has completed preloading and is capable of performing image processing.

[0068] In some embodiments, during the process of controlling the first camera and the second camera to acquire raw images respectively, the image processing program can be preloaded.

[0069] For example, to facilitate understanding, it will be done through Figure 3C The illustrated embodiment provides an exemplary description of the above-described photographing method. Figure 3C This is a schematic diagram illustrating the preloading of an image processing program according to an exemplary embodiment. For example... Figure 3C As shown, during the image acquisition stage, the image processing program is preloaded (e.g., preload the first image processing program, and / or preload the second image processing program, etc.), and in the post-processing stage following the image acquisition stage, the corresponding image processing program is called (e.g., call the first image processing program, and / or call the second image processing program, etc.). It is important to understand that in... Figure 3C In this embodiment, the image processing program calls performed in the post-processing stage are different from... Figure 3A The image processing program call in the embodiment, Figure 3C The image processing program invoked in this embodiment is a pre-loaded image processing program. Therefore, in Figure 3C In the example, the image processing program is not loaded during the post-processing stage when the image processing program is invoked.

[0070] In some embodiments, the preloaded image processing program may be an image processing program that is pre-called during image processing, such as an application (or algorithm) for HDR processing, an application (or algorithm) for depth calculation, an application (or algorithm) for portrait processing, etc.

[0071] It should be noted that the pre-called image processing program can perform corresponding image processing on the file formats corresponding to the first and second images. For example, the pre-called image processing program can perform corresponding image processing on RAW images.

[0072] Based on this, the image processing program can be preloaded, thereby reducing the waiting time for taking pictures by reducing the waiting time for the image processing program to load during the image post-processing stage.

[0073] It is clear that, taking the portrait mode as an example, in the post-processing stage of the above embodiment, the image processing program takes the input image to be processed as a RAW image (i.e., both the first and second images are RAW images) and performs corresponding image processing on the image to be processed to obtain the captured image. The corresponding image processing process can be, for example, as follows: Figure 4 As shown in the relevant embodiments. For ease of understanding, the following embodiments will describe a scenario where the first camera is the main camera, the first image is an image acquired by the main camera, the second camera is the secondary camera, and the second image is an image acquired by the secondary camera.

[0074] Figure 4This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 4 As shown, the method includes the following steps.

[0075] In step S31, the disparity values ​​of the first image and the second image are determined, and the depth map of the first image is determined based on the disparity values, wherein the disparity values ​​characterize the degree of positional difference between matching pixel pairs in the first image and the second image.

[0076] In step S32, the first image is used as the reference frame, and the HDR image and the depth map are fused to obtain the captured image.

[0077] In this embodiment of the disclosure, by performing corresponding processing on the first image and / or the second image, an HDR image of the first image and a depth map corresponding to the first image are obtained respectively. Then, by performing image fusion using the obtained HDR image and depth map, a portrait mode photographed image can be obtained.

[0078] It is understandable that in portrait mode, the image captured by the main camera contains the target object, and the image captured by the secondary camera is used to assist the image captured by the main camera in determining the depth information corresponding to the target object. This enables the blurring of areas that do not contain the target object (such as the background area) to obtain the image captured in portrait mode.

[0079] Therefore, if the first camera is a secondary camera, the first image is an image captured by the secondary camera, and the second camera is the primary camera, the second image is an image captured by the primary camera, then the scene can be based on... Figure 4 The processing steps described in the relevant embodiments are adapted to the objects to be processed, and will not be repeated here.

[0080] It should be noted that parallax value can refer, for example, to the difference in pixels along the horizontal direction between images of the same object viewed from two different perspectives. Images of the same object viewed from two different perspectives can be, for example, the same pixel points (i.e., pixel pairs) in the first image and the second image.

[0081] For example, disparity values ​​can be obtained in the following way:

[0082] (1) Feature matching: Find corresponding feature points in the images captured by the two cameras (e.g., the first image and the second image) and match them.

[0083] (2) Disparity calculation: For a pair of matched feature points, calculate the difference in their horizontal positions in the two images. This difference is the disparity value.

[0084] For example, Figure 5 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 5 As shown, the method includes the following steps.

[0085] In step S41, based on the HDR image and the first image, a brightening template corresponding to the first image is determined. The brightening template is used to adjust the dynamic range corresponding to the first image.

[0086] In step S42, the brightening template and the depth map are image fused.

[0087] In this embodiment, a portrait mode image is obtained by using a highlighting template corresponding to the first image and a depth map for image fusion. The dynamic range of the first image can be adjusted based on the highlighting template during multiple image fusion processes, reducing the number of times the HDR algorithm is called to solve for the HDR image of the first image, thereby reducing processing time and consequently reducing capture time.

[0088] In some embodiments, the brightening template can be pre-constructed based on the first acquired image and directly invoked during image post-processing. Alternatively, it can be constructed using the first acquired image during the initial photo capture and invoked during subsequent photo captures.

[0089] In some embodiments, the construction of the highlighting template can be achieved, for example, in the following way:

[0090] (1) Capture HDR images: Take multiple images of the same scene at different exposures to capture all the details from the darkest to the brightest.

[0091] (2) Preprocessing: Perform preliminary processing on the acquired images, such as noise reduction and white balance correction.

[0092] (3) Feature extraction and matching: Find the corresponding feature points in multiple images and perform matching.

[0093] (4) Disparity calculation: Calculate the disparity map. Disparity is the difference in the horizontal position of the same feature point in two images.

[0094] (5) Depth map generation: Generate a depth map based on the disparity map. The depth map represents the distance between each pixel and the camera.

[0095] (6) Tone mapping: Mapping the wide dynamic range of an HDR image to the limited dynamic range of a standard display device to generate an LDR image.

[0096] (7) Analyze brightness distribution: Use image processing technology to analyze the brightness distribution of LDR images and identify areas that need to be brightened.

[0097] (8) Generate a brightening template: Based on the brightness distribution analysis results, a brightening template is generated, which defines the areas that need to be brightened and the degree of brightening.

[0098] It is understandable that the highlighting template can be constructed in other ways, which will not be elaborated here.

[0099] Based on this, a highlighting template can be constructed, and the resulting highlighting template can be output to an image processing program for portrait processing.

[0100] For obtaining depth maps, for example, it can be done through methods such as... Figure 6A The implementation method is as described in the relevant embodiments. Figure 6A This is a flowchart illustrating a method for determining a depth map of a first image according to an exemplary embodiment. Figure 6A As shown, it includes the following steps.

[0101] In step S51, a target mapping relationship is established, which is the mapping relationship between disparity value and depth information.

[0102] In step S52, based on pixel pairs, the disparity values ​​corresponding to pixel pairs, and the target mapping relationship, the depth information corresponding to the pixels in the first image in the pixel pair is determined.

[0103] In step S53, the depth map corresponding to the first image is determined based on the depth information.

[0104] In this embodiment of the disclosure, the depth information corresponding to the pixels in the first image is determined by establishing a mapping relationship between disparity values ​​and depth information, and then the depth map corresponding to the first image is determined based on the depth information corresponding to the pixels in the first image.

[0105] In some embodiments, the target mapping relationship can be, for example, in the following form:

[0106] Depth=(f*baseline) / disparity.

[0107] Where f is the focal length of the camera, baseline is the baseline distance between the two cameras, and disparity is the parallax value.

[0108] The focal length of the camera can be obtained through camera-related parameters, as can the baseline distance between two cameras. The parallax value can be obtained, for example, through... Figure 4 The implementation details and related embodiments are not elaborated here.

[0109] Based on this, the depth map of the first image can be determined, and then the determined depth map can be sent to the image processing program for portrait processing to perform the corresponding portrait processing.

[0110] In some embodiments, after obtaining the photographed image, the image can be format-converted based on set requirements (e.g., converted from RAW to YUV), and the converted image can be used as a preview image.

[0111] For ease of understanding, the embodiments disclosed herein will be explained through... Figure 6B An exemplary description of the complete post-processing stage is provided. Figure 6B This is a schematic diagram illustrating an image post-processing stage according to an exemplary embodiment. For example... Figure 6B As shown, continuing Figure 2B In a related embodiment, the image processing program for post-processing obtains RAW1 and RAW2 images. For the RAW1 image, a corresponding frame selection algorithm can be used to determine the image to be used for subsequent image processing operations. The image determined by the frame selection algorithm is then input into the image processing program for HDR processing to obtain the corresponding HDR image and / or brightening template, such as the first image processing program (the corresponding process can be found in...). Figure 4 , and / or Figure 5 The relevant embodiments are described in detail here, and the image selected by the frame selection algorithm from the RAW1 image, combined with the image selected from the RAW2 image by the relay node, is input into the image processing program for depth calculation, such as the second image processing program, to obtain the depth map corresponding to the RAW1 image (the corresponding process can be found in...). Figure 6A (As described in the relevant embodiments, they will not be repeated here.) The depth map, HDR image, and / or brightening template are input into an image processing program for portrait processing, such as a third image processing program, and the RAW1 image is used as the reference frame for image fusion to obtain a RAW format portrait image. The RAW format is then converted to YUV format, and the image after format conversion is output as a preview image.

[0112] In summary, by acquiring images of the same format during the image acquisition stage through the above embodiments, the subsequent image processing stage saves time on format conversion. Furthermore, by optimizing the processing process (e.g., preloading the image processing program) and using a brightening template for high dynamic range adjustment during the image processing stage, the processing time is also reduced to some extent, thereby reducing the overall shooting time. This allows the terminal to process the user's shooting needs more quickly and improves the user's shooting experience.

[0113] Based on the same concept, this disclosure also provides a photographing device.

[0114] It is understood that the photographing device provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0115] Figure 7 This is a block diagram illustrating a photographing device according to an exemplary embodiment. (Refer to...) Figure 7 The device 100 includes a control unit 101 and a processing unit 102.

[0116] The control unit 101 is used to control the first camera and the second camera to acquire raw images respectively when the camera application is in shooting mode, so as to obtain the first image and the second image, wherein the raw image is an image in RAW data format.

[0117] The processing unit 102 is used to adjust the dynamic range of the first image to obtain a high dynamic range (HDR) image of the first image, and to perform image fusion based on the HDR image, the first image, and the second image to obtain a captured image.

[0118] In some embodiments, the processing unit 102 performs image fusion based on the HDR image, the first image, and the second image to obtain a captured image in the following manner: the dynamic range of the first image is adjusted to obtain an HDR image of the first image; the disparity values ​​of the first image and the second image are determined, and the depth map of the first image is determined based on the disparity values, wherein the disparity values ​​characterize the degree of positional difference between matching pixel pairs in the first image and the second image; the first image is used as a reference frame, and the HDR image and the depth map are fused to obtain a captured image.

[0119] In some embodiments, the processing unit 102 performs image fusion of the HDR image and the depth map in the following manner: based on the HDR image and the first image, a brightening template corresponding to the first image is determined, the brightening template is used to adjust the dynamic range corresponding to the first image; and the brightening template and the depth map are fused.

[0120] In some embodiments, the processing unit 102 determines the depth map of the first image based on the disparity value in the following manner: establishing a target mapping relationship, which is a mapping relationship between disparity value and depth information; determining the depth information corresponding to the pixel in the first image in the pixel pair based on the pixel pair, the disparity value corresponding to the pixel pair, and the target mapping relationship; and determining the depth map corresponding to the first image based on the depth information.

[0121] In some embodiments, the processing unit 102 obtains the photographed image by calling a preloaded image processing program to perform image processing on the first image and the second image to obtain the photographed image. The image processing program is used to perform image processing on the original image to obtain the photographed image.

[0122] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0123] Figure 8 This is a block diagram illustrating a device for taking a photograph according to an exemplary embodiment. The device 200 can be provided as a terminal. For example, the device 200 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0124] Reference Figure 8 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0125] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0126] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0127] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0128] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0129] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0130] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0131] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0132] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0133] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0134] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0135] Figure 9 This is a block diagram illustrating an apparatus for taking a photograph according to an exemplary embodiment. For example, apparatus 300 may be provided as a server. (See also...) Figure 9The device 300 includes a processing component 322, which further includes one or more processors, and memory resources represented by memory 332 for storing instructions, such as application programs, that can be executed by the processing component 322. The application programs stored in memory 332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 322 is configured to execute instructions to perform the aforementioned photographing method.

[0136] Device 300 may also include a power supply component 326 configured to perform power management of device 300, a wired or wireless network interface 350 configured to connect device 300 to a network, and an input / output (I / O) interface 358. Device 300 may operate on an operating system stored in memory 332, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0137] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0138] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0139] It is further understood that the terms “center,” “longitudinal,” “lateral,” “front,” “rear,” “up,” “down,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” and “outer,” etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation.

[0140] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0141] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0142] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

Claims

1. A method for taking photos, characterized in that, include: In response to the camera application being in shooting mode, the first camera and the second camera are controlled to acquire raw images respectively, resulting in a first image and a second image, wherein the raw image is an image in RAW data format; The dynamic range of the first image is adjusted to obtain a high dynamic range (HDR) image of the first image; The captured image is obtained by fusing the HDR image, the first image, and the second image.

2. The method according to claim 1, characterized in that, The process of fusing the HDR image, the first image, and the second image to obtain the captured image includes: Determine the disparity values ​​of the first image and the second image, and determine the depth map of the first image based on the disparity values, wherein the disparity values ​​characterize the degree of positional difference between matching pixel pairs in the first image and the second image; The first image is used as a reference frame, and the HDR image and the depth map are fused to obtain the captured image.

3. The method according to claim 2, characterized in that, The step of fusing the HDR image and the depth map includes: Based on the HDR image and the first image, a brightening template corresponding to the first image is determined, and the brightening template is used to adjust the dynamic range corresponding to the first image. The highlighting template and the depth map are then image-fused.

4. The method according to claim 2, characterized in that, Determining the depth map of the first image based on the disparity value includes: Establish a target mapping relationship, which is the mapping relationship between the disparity value and the depth information; Based on the pixel pair, the disparity value corresponding to the pixel pair, and the target mapping relationship, the depth information corresponding to the pixel in the first image in the pixel pair is determined; The depth map corresponding to the first image is determined based on the depth information.

5. The method according to claim 1, characterized in that, The photographed image was obtained based on the following method: A preloaded image processing program is invoked to process the first image and the second image to obtain the captured image. The image processing program is used to process the original image to obtain the captured image.

6. A photographing device, characterized in that, include: The control unit is used to control the first camera and the second camera to acquire raw images respectively when the camera application is in shooting mode, so as to obtain the first image and the second image, wherein the raw image is an image in RAW data format; The processing unit is configured to perform dynamic range adjustment on the first image to obtain a high dynamic range (HDR) image of the first image, and perform image fusion based on the HDR image, the first image, and the second image to obtain a captured image.

7. The apparatus according to claim 6, characterized in that, The processing unit performs image fusion based on the HDR image, the first image, and the second image to obtain the captured image in the following manner: Determine the disparity values ​​of the first image and the second image, and determine the depth map of the first image based on the disparity values, wherein the disparity values ​​characterize the degree of positional difference between matching pixel pairs in the first image and the second image; The first image is used as a reference frame, and the HDR image and the depth map are fused to obtain the captured image.

8. The apparatus according to claim 7, characterized in that, The processing unit performs image fusion of the HDR image and the depth map in the following manner: Based on the HDR image and the first image, a brightening template corresponding to the first image is determined, and the brightening template is used to adjust the dynamic range corresponding to the first image. The highlighting template and the depth map are then image-fused.

9. The apparatus according to claim 7, characterized in that, The processing unit determines the depth map of the first image based on the disparity value in the following manner: Establish a target mapping relationship, which is the mapping relationship between the disparity value and the depth information; Based on the pixel pair, the disparity value corresponding to the pixel pair, and the target mapping relationship, the depth information corresponding to the pixel in the first image in the pixel pair is determined; The depth map corresponding to the first image is determined based on the depth information.

10. The apparatus according to claim 6, characterized in that, The processing unit obtains the captured image in the following manner: A preloaded image processing program is invoked to process the first image and the second image to obtain the captured image. The image processing program is used to process the original image to obtain the captured image.

11. An electronic device, characterized in that, include: processor: Memory used to store processor-executable instructions; The processor is configured to perform the photographing method according to any one of claims 1 to 5.

12. A storage medium, characterized in that, The storage medium stores instructions that, when executed by a processor, enable the processor to perform the photographing method according to any one of claims 1 to 5.

Images