Image signal processing in multi-camera systems

Abstract

A processor generates a depth map from two images, including high dynamic range (HDR) images. The processor receives one or more first images from a first image sensor, where the one or more first images have a first field of view (FOV) and a first resolution, and receives one or more second images from a second image sensor, where the one or more second images have a second FOV and a second resolution. The processor generates a first HDR image from the one or more first images, and performs tone alignment on the one or more second images based on the first HDR image to generate a second HDR image. The processor generates a depth map using the first HDR image and the second HDR image. The processor can use the depth map to apply a bokeh effect to the first HDR image.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Application No. 17 / 029,526, filed on September 23, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to image capture and processing. Background Technology

[0004] Image capture devices are incorporated into a wide variety of devices. In this disclosure, an image capture device refers to any device capable of capturing one or more digital images, including devices capable of capturing still images and devices capable of capturing sequences of images to record video. As examples, an image capture device may include a standalone digital camera or digital video camcorder, a wireless communication device equipped with a camera, a handheld device (such as a mobile phone with one or more cameras, a cellular or satellite radio phone, a personal digital assistant (PDA) equipped with a camera, a panel or tablet computer, a gaming device), a computer device including a camera (such as a so-called "webcam"), or any device with digital image or video capabilities.

[0005] Images captured by digital cameras are typically low dynamic range (LDR) images (sometimes referred to as standard dynamic range or SDR). Depending on the camera's exposure settings, images captured in LDR / SDR may include loss of detail in either bright or dark areas of the image under certain lighting conditions. High dynamic range (HDR) images can represent dark areas (e.g., shadows) and bright areas (e.g., sunlight) more accurately because HDR images generally include more possible levels of brightness (e.g., luminance). In some examples, an HDR image can be generated by combining multiple LDR images at different exposure settings.

[0006] Image capture devices can include multiple image sensors and / or multiple lenses, which can be used to support various imaging technologies such as high dynamic range imaging (HDRI), multi-frame HDRI (MFHDRI), etc. The image sensors can then forward image data to a camera processor. Example lens types include wide-angle lenses, ultra-wide-angle lenses, telephoto lenses, telephoto lenses, periscope zoom lenses, fisheye lenses, macro lenses, main lenses, or various combinations thereof. As an example, an image capture device with a dual-camera configuration can include a wide-angle lens and a telephoto lens. Similarly, in addition to wide-angle and telephoto lenses, a triple-camera configuration can also include an ultra-wide-angle lens. By using multiple lenses and / or image sensors, the camera processor of the image capture device can capture images with various fields of view (FOV), adjust zoom levels, apply focus effects, and / or create one or more composite frames of image data (e.g., images generated from a combination of multiple images).

[0007] Some example image processing techniques rely on image segmentation algorithms that divide an image into multiple segments, which can be analyzed or processed to produce specific image effects. Some practical applications of image segmentation include, but are not limited to, digital depth-of-field effects (e.g., "defocus" or "portrait mode" effects), chroma key synthesis, feature extraction, recognition tasks (e.g., object and face recognition), machine vision, and medical imaging. However, current image segmentation techniques often produce poor segmentation results and, in many cases, are only applicable to specific types of images. Summary of the Invention

[0008] Generally, this disclosure describes camera processing techniques relating to high dynamic range (HDR) imaging and / or depth map computation for segmentation-based image effects such as depth-of-field effects (e.g., defocusing effects). Generally, image segmentation is the process of dividing an image into multiple segments (e.g., sets of pixels and / or features representing the image). In some examples, image segmentation can be used to identify the location and boundaries of objects and other features in an image. Segmentation-based image effects can include any image effect that applies such an effect to certain portions or features of an image. In some examples, portions or features of an image can be segmented based on depth values ​​determined for these portions of the image's features. That is, different regions or features of an image can be associated with different depth values.

[0009] In some example techniques, when applying a defocusing effect to an HDR image, the imaging device can be configured to use a primary camera (e.g., a master camera) to output the HDR image and a secondary camera (e.g., a slave camera) to output a non-HDR image. The non-HDR image, along with the HDR image, is used to calculate a depth map for use in the defocusing effect. However, because the secondary camera is capturing the same HDR scene as the primary camera, the non-HDR image may be underexposed or overexposed. Therefore, the non-HDR image output from the secondary camera may exhibit considerable loss of detail relative to the HDR image generated from the primary camera. Consequently, any depth map calculated from such an image may lack accuracy, thus affecting the quality of any defocusing effect applied based on such a depth map. Similarly, such a depth map may also lack the accuracy of other image effects based on image segmentation.

[0010] The aforementioned problems can be solved by the HDR synchronization techniques described in this disclosure. As explained in more detail below, the camera processor of the imaging device can be configured to determine the field of view (FOV) and resolution of the image output by the primary camera. The camera processor can process the image output by the primary camera to produce an HDR image. The camera processor can instruct the secondary camera to synchronize both the FOV and resolution of the image output by the secondary camera to be the same as the video FOV and resolution of the image output by the primary camera. The camera processor can then perform tone alignment on the image output by the secondary camera to match the tonal range present in the HDR image generated from the image output by the primary camera.

[0011] In one example, the camera processor can achieve tone alignment by performing the same HDR processing on images received from the secondary camera(s) as on images received from the primary camera. In another example, the camera can perform tone alignment on images received from the secondary camera using a different HDR technique, where the different HDR technique approximates the tonal range produced by the HDR method used on the image output from the primary camera. In this way, the camera processor can use two HDR images with respect to FOV, resolution, and tone synchronization to compute a depth map, thereby improving the accuracy of the depth map. A more accurate depth map can then improve the quality of any image effects, such as depth-of-field effects (including defocusing), applied to the image based on image segmentation using the computed depth map.

[0012] In one example, the technology disclosed herein relates to a camera processing method, the method comprising receiving one or more first images from a first image sensor, wherein the one or more first images have a first field of view (FOV) and a first resolution, generating a first high dynamic range (HDR) image from the one or more first images, generating a second HDR image, and generating a depth map using the first HDR image and the second HDR image.

[0013] In another example, the technology disclosed herein relates to a device configured for camera processing, the device including a memory configured to receive images and one or more processors in communication with the memory, the one or more processors being configured to receive one or more first images from a first image sensor, wherein the one or more first images have a first FOV and a first resolution, generate a first HDR image from the one or more first images, generate a second HDR image, and use the first HDR and the second HDR image to generate a depth map.

[0014] This disclosure also describes components for performing any of the techniques described herein. In one example, the technology of this disclosure relates to an apparatus configured to perform camera processing, the apparatus including components for receiving one or more first images from a first image sensor, wherein the one or more first images have a first FOV and a first resolution, components for generating a first HDR image from the one or more first images, components for generating a second HDR image, and components for generating a depth map using the first HDR image and the second HDR image.

[0015] In another example, this disclosure describes a non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to receive one or more first images from a first image sensor, wherein the one or more first images have a first FOV and a first resolution, generate a first HDR image from the one or more first images, generate a second HDR image, and generate a depth map using the first HDR image and the second HDR image.

[0016] This invention is intended to provide an overview of the subject matter described herein. It is not intended to provide an exclusive or exhaustive interpretation of the systems, apparatus, and methods detailed in the accompanying drawings and description. Further details of one or more examples of the disclosed technology are set forth in the drawings and the description below. Other features, objects, and advantages of the disclosed technology will become apparent from the specification, drawings, and claims. Attached Figure Description

[0017] Figure 1 It is a block diagram of a device configured to perform one or more example technologies described in this disclosure.

[0018] Figure 2 A more detailed map is shown. Figure 1 A block diagram of example components of a computing device.

[0019] Figure 3 This is a conceptual diagram illustrating an example of field-of-view synchronization.

[0020] Figure 4 This is a block diagram illustrating example field-of-view synchronization techniques using wide-angle and ultra-wide-angle fields of view.

[0021] Figure 5 The figure illustrates an example field-of-view and resolution synchronization technique of this disclosure.

[0022] Figure 6 The figure illustrates an example resolution synchronization technique of this disclosure.

[0023] Figure 7 The figure illustrates another example field-of-view and resolution synchronization technique disclosed herein for the Quad Bayer Color Filter Array (QCFA).

[0024] Figure 8 This is a flowchart illustrating an example operation of camera processing according to the example technology of this disclosure.

[0025] Figure 9 This is a flowchart illustrating another example operation of camera processing according to the example technology of this disclosure.

[0026] Throughout the description and accompanying drawings, the same reference numerals denote the same elements. Detailed Implementation

[0027] In some digital camera applications, depth maps are used to perform one or more image effects (such as depth-of-field effects) on an image based on image segmentation. Broadly speaking, image segmentation is the process of dividing an image into multiple segments (e.g., sets of pixels and / or features representing the image). In some examples, image segmentation can be used to identify the location and boundaries of objects and other features in an image. Segmentation-based image effects can include any image effect in which such an effect is applied to certain parts or features of an image. In some examples, parts or features of an image can be segmented based on depth values ​​determined for parts of such image features. That is, different regions or features of an image can be associated with different depth values.

[0028] A depth map includes depth values ​​that represent the distance of each pixel, region, and / or feature of an image relative to the image sensor, or the distance of each pixel, region, or feature of an image relative to the focal point. In some examples, a depth map may include depth values ​​relative to a fixed point (e.g., a camera sensor). In other examples, a depth map may include disparity values ​​associated with depth. For example, disparity values ​​may indicate the relative distances between features in an image. Disparity values ​​can be used to determine a final depth value relative to some spatial offset.

[0029] An example of depth-of-field effect is defocus. Defocus is defined as the effect of a soft, out-of-focus background achieved when photographing a subject in the foreground (e.g., a portrait subject). In traditional digital SLR (DSLR) cameras, different defocus effects can be achieved by using different lenses with different fields of view and lens geometries. When using a telephoto lens with approximately 2x zoom, defocus is often aesthetically pleasing for portraits.

[0030] Defocusing effects can also be achieved using post-processing and depth maps on devices with small shape factors (e.g., mobile phones), even if the camera lenses on such devices do not inherently produce defocusing. For example, a mobile device may include multiple cameras, and two different cameras may be used simultaneously or nearly simultaneously to capture images. Because the two cameras are separated by a certain physical distance, the depth (e.g., distance from the camera sensor) of features in the first image can be calculated based on a comparison of features in the first image and features in the second image. Defocusing processing can then use the calculated depth map to blur pixels in the image. For example, pixels and / or features at depths matching the image's focus distance remain in focus, while pixels at greater focus distances are blurred. In one example, pixels and / or features at depths not at the same focus distance are blurred at the same rate. In another example, the farther a pixel and / or feature is from the focus distance, the more blur can be applied by the defocusing effect.

[0031] In some examples, it is desirable to apply defocusing effects or other image effects based on image segmentation to high dynamic range (HDR) images. HDR images are generally generated in such a way that a wide range of brightness levels (e.g., light levels) are produced from a scene with a relatively high level of detail. In most imaging devices, the exposure of light applied to the image sensor can be changed in one of two ways: increasing / decreasing the aperture size or increasing / decreasing the duration of each exposure (e.g., by adjusting the shutter speed). Due to the physical characteristics of camera sensors, it may be impossible to capture all the brightness levels present in a scene with a very wide range of brightness levels. That is, a camera sensor may not be able to record all the brightness levels in a scene that are perceptible to the human eye.

[0032] For example, to preserve detail in darker areas of a scene, a camera might decrease the shutter speed and / or increase the aperture. However, this would result in overexposure of brighter areas (e.g., the image sensor output would show the maximum brightness level for many different actual brightness levels), and detail in those brighter areas would be lost. Conversely, to preserve detail in brighter areas, a camera might increase the shutter speed and / or decrease the aperture. However, this would result in underexposure of darker areas (e.g., the image sensor output would show the minimum brightness level for many different actual brightness levels), and detail in those darker areas would be lost.

[0033] In some examples, the camera can be configured to generate an HDR image by capturing multiple images with varying exposure settings and then combining them to form a final image that more accurately represents all the brightness levels seen. In some examples, the exposure variations used for HDR generation are achieved simply by changing the exposure time rather than the aperture size. This is because changing the aperture size can also affect the depth of field of each image in the output image, which may be undesirable in some applications. In other applications, the camera can adjust both the aperture and shutter speed to output multiple images used to form the HDR image.

[0034] However, in some example techniques, when applying defocusing effects or other image segmentation-based effects to an HDR image, the imaging device can be configured to use a primary camera (e.g., the first camera) to generate the HDR image and a secondary camera (e.g., the second camera) to generate another non-HDR image. The non-HDR image is used together with the HDR image to calculate the depth map used in the defocusing effect. However, the non-HDR image may be underexposed or overexposed because the secondary camera is capturing the same HDR scene as the primary camera. Therefore, the non-HDR image generated by the secondary camera may exhibit considerable loss of detail compared to the HDR image generated by the image generated by the primary camera. Consequently, any depth map calculated from such an image may lack accuracy, thus affecting the quality of any defocusing effect applied based on such a depth map.

[0035] The aforementioned problems can be solved by the HDR synchronization techniques described in this disclosure. As will be explained in more detail below, the camera processor of the imaging device can be configured to determine the field of view (FOV) and resolution of the image output by the primary camera. The camera processor can process the image received from the primary camera to generate an HDR image. The camera processor can synchronize both the FOV and resolution of the image output by the secondary camera to be the same as those of the image output by the primary camera. The camera processor can also generate an HDR image based on the output of the secondary camera. For example, the camera processor can perform tone alignment on the image output by the secondary camera to match the tonal range presented in the HDR image generated based on the image output by the primary camera.

[0036] In one example, the camera processor can achieve tone alignment by performing the same HDR processing on the image received from the secondary camera as it does on the image received from the primary camera. In another example, the camera can perform tone alignment on the image received from the secondary camera using a different HDR technique, where the different HDR technique approximates the tonal range produced by the HDR technique used on the image output from the primary camera. In this way, the camera processor can use two HDR images synchronized in FOV, resolution, and tone to calculate a depth map, thereby improving the accuracy of the depth map. A more accurate depth map can then be used to improve the quality of any image effects, such as depth-of-field effects, including defocusing effects applied to the image using the calculated depth map, based on image segmentation.

[0037] Figure 1This is a block diagram of a computing device 10 configured to perform one or more example techniques described in this disclosure for calculating a depth map from an HDR image and for applying image effects to the HDR image based on the calculated depth map. Examples of computing devices 10 include computers (e.g., personal computers, desktop computers, or laptop computers), mobile devices such as portable tablet computers, wireless communication devices (e.g., mobile TVs, cellular phones, satellite phones, and / or mobile TV phones), internet TVs, digital cameras, digital video recorders, handheld devices such as portable video game devices or personal digital assistants (PDAs), drone devices, or any device that may include one or more cameras. In some examples, computing device 10 may include one or more camera processors 14, a central processing unit (CPU) 16, a video encoder / decoder 17, a graphics processing unit (GPU) 18, local memory 20 of the GPU 18, a user interface 22, a memory controller 24 providing access to system memory 30, and a display interface 26 outputting signals that cause graphic data to be displayed on a display 28.

[0038] like Figure 1 As illustrated in the example, computing device 10 includes one or more image sensors 12A-N. Image sensors 12A-N may be simply referred to as "sensor 12" in some instances herein, while in other instances, they may be referred to as multiple "sensors 12" where appropriate. Sensor 12 can be any type of image sensor, including sensors that include Bayer filters or high dynamic range (HDR) interlaced sensors (such as Quad Bayer sensors).

[0039] The computing device 10 also includes one or more lenses 13A-N. Similarly, lenses 13A-N may be simply referred to as "lens 13" in some cases herein, while in others, they may be referred to as multiple "lenses 13" where appropriate. In some examples, sensor 12 refers to one or more image sensors 12, each image sensor 12 may each include processing circuitry, an array of pixel sensors (e.g., pixels) for capturing light representations, and memory (such as buffer memory or on-chip sensor memory). In some examples, each of the image sensors 12 may be coupled to a different type of lens 13, with each lens and image sensor combination having a different aperture and / or field of view. Example lenses may include telephoto lenses, wide-angle lenses, ultra-wide-angle lenses, or other lens types.

[0040] like Figure 1As shown, computing device 10 includes a plurality of cameras 15. As used herein, the term “camera” refers to a specific image sensor 12 of computing device 10 or a plurality of image sensors 12 in computing device 10, wherein the image sensors 12 are arranged in combination with one or more lenses 13 on computing device 10. That is, a first camera 15 of computing device 10 refers to a first aggregate device including one or more image sensors 12 and one or more lenses 13, and a second camera 15 separate from the first camera 15 refers to a second aggregate device including one or more image sensors 12 and one or more lenses 13. Furthermore, image data may be received from the image sensors 12 of the specific camera 15 by camera processor 14 or CPU 16. That is, in some examples, camera processor 14 or CPU 16 may receive a first set of image data frames from the first image sensor 12 of the first camera 15 and a second set of image data from the second image sensor 12 of the second camera 15.

[0041] In one example, the term "camera" as used herein refers to a combined image sensor 12 and lens 13, coupled together, configured to capture at least one frame of image data and to forward that at least one frame of image data to a camera processor 14 and / or a CPU 16. In an illustrative example, a first camera 15 is configured to forward the first frame of image data to the camera processor 14, and a second camera 15 is configured to forward a second frame of image data to the camera processor 14, wherein the two frames are captured by different cameras, as can be demonstrated, for example, by the difference in the FOV and / or zoom level between the first and second frames. The difference in FOV and / or zoom level may correspond to a difference in focal length between the first camera 15 and the second camera 15.

[0042] The computing device 10 may include a dual-lens device, a triple-lens device, a 360-degree camera lens device, etc. Therefore, the combination of each lens 13 and image sensor 12 can provide various zoom levels, angle of view (AOV), focal length, field of view (FOV), etc. In some examples, a specific image sensor 12 can be assigned to each lens 13, and vice versa. For example, multiple image sensors 12 can be assigned to different lens types (e.g., wide-angle lens, ultra-wide-angle lens, telephoto lens, and / or periscope lens, etc.).

[0043] Camera processor 14 can be configured to control the operation of camera 15 and perform processing on images received from camera 15. In some examples, camera processor 14 may include image signal processor (ISP) 23. For example, camera processor 14 may include circuitry for processing image data. Camera processor 14 (including ISP 23) can be configured to perform various operations on image data captured by image sensor 12, including automatic white balance, color correction, or other post-processing operations. Figure 1 A single ISP 23 configured to operate on the output of camera 15 is shown. In other examples, camera processor 14 may include an ISP 23 for each camera in camera 15 to improve processing speed and / or improve synchronization for simultaneous image capture from multiple cameras of camera 15.

[0044] In some examples, camera processor 14 can perform "3A". This algorithm may include autofocus (AF), auto exposure control (AEC), and auto white balance (AWB) techniques. In such examples, 3A can represent the functionality of a statistical algorithm processing engine, in which one of the camera processors 14 can implement and operate such a processing engine.

[0045] In some examples, camera processor 14 is configured to receive image frames (e.g., pixel data) from image sensor 12 and process these image frames to generate image and / or video content. For example, image sensor 12 may be configured to capture individual frames, frame bursts, frame sequences for generating video content, still photographs captured while recording video, preview frames, or moving images from before and / or after capturing still photographs. CPU 16, GPU 18, camera processor 14, or some other circuitry may be configured to process the images and / or video content captured by sensor 12 into images or videos for display on display 28. Image frames can generally refer to data frames of still images or video data frames or combinations thereof, such as photographs with motion. Camera processor 14 can receive pixel data of image frames from sensor 12 in any format. For example, pixel data may include different color formats such as RGB, YCbCr, YUV, etc. In any case, camera processor 14 can receive multiple frames of image data from image sensor 12.

[0046] In examples including multiple camera processors 14, the camera processors 14 may share the sensor 12, with each of the camera processors 14 interfaced with each of the sensors 12. In any case, the camera processors 14 may use multiple pixel sensors of the sensor 12 to initiate the capture of video or images of the scene. In some examples, the video may include a sequence of individual frames. Thus, the camera processors 14 cause the sensor 12 to use multiple pixel sensors to capture images. The sensor 12 may then output pixel information (e.g., pixel values, luminance values, color values, charge values, analog-to-digital unit (ADU) values, etc.) to the camera processors 14, the pixel information representing the captured image or a sequence of captured images. In some examples, the camera processors 14 may process monochrome and / or color images to obtain an enhanced color image of the scene. In some examples, the camera processor 14 may determine a general blending weight coefficient for blending different types of pixels, or may determine different blending weight coefficients for blending different types of pixels that constitute a pixel frame (e.g., a first blending weight coefficient for blending pixels obtained via the monochrome sensor of the first camera 15 and an image obtained via the monochrome sensor of the second camera 15, a second blending weight coefficient for blending pixels obtained via the Bayer sensor of the first camera 15 and pixels obtained via the Bayer sensor of the second camera 15, etc.).

[0047] In the examples disclosed herein, camera processor 14 can cause a specific image sensor in image sensor 12 to generate images in HDR mode. In one example, camera processor 14 can cause image sensor 12 to generate multiple images with different exposure settings. In another example, camera processor 14 can be configured to cause image sensor 12 to generate multiple images with different shutter speeds but the same aperture size. Image sensor 12 can output multiple images with a specific field of view (FOV) and a specific resolution, for example, based on zoom settings and the selected lens type (e.g., telephoto, wide-angle, ultra-wide-angle, etc.). ISP 23 can receive these multiple images and, in addition to any 3A or other processing, can perform multi-frame HDR processing techniques to combine the multiple images into an HDR image.

[0048] In some examples, such as those indicated by user selection, camera processor 14 can also perform image segmentation-based image effects on HDR images, such as defocus effects. A defocus effect can be defined as the effect of a soft, out-of-focus background achieved when shooting a subject in the foreground (e.g., a portrait). In digital single-lens reflex (DSLR) cameras, different defocus effects can be achieved by using different lenses with different fields of view and lens geometries. For portraits, a defocus effect is often aesthetically pleasing when using a telephoto lens with approximately 2x zoom.

[0049] Defocusing effects can also be achieved by camera processor 14 through post-processing using depth maps. For example, camera processor 15 can instruct camera 15 to generate two images simultaneously using two image sensors 12. Because the two image sensors 12 of camera 15 are separated by a physical distance, the depth (e.g., distance from image sensor 12) of features from the first image from the first image sensor 12A (e.g., the primary image sensor or "first" image sensor) can be calculated based on a comparison with these features in the second image generated by the second image sensor 12B (e.g., the secondary image sensor or "secondary" image sensor). Camera processor 14 can then apply a defocusing effect to the HDR image generated from the first image sensor 12A by blurring features in the image using the calculated depth map. For example, camera processor 14 can keep the focus features at a depth matching the focus distance of the generated image, while blurring features further away from the focus distance. In some examples, the farther a pixel is from the focal length, the more blurred the defocusing effect can be applied by camera processor 14.

[0050] In some exemplary techniques, when applying image segmentation-based image effects to an HDR image, the imaging device can be configured to capture the HDR image using a primary camera (e.g., a first camera) and another non-HDR image using a secondary camera (e.g., a second camera). The non-HDR image, along with the HDR image, is used to calculate a depth map for use in the image effects. However, the non-HDR image may be underexposed or overexposed because the secondary camera is capturing the same HDR scene as the primary camera. Therefore, the non-HDR image captured by the secondary camera may exhibit considerable loss of detail compared to the HDR image captured by the primary camera. Thus, any depth map calculated from such an image may lack accuracy, thereby affecting the quality of any image effects applied based on such a depth map.

[0051] The aforementioned problems can be solved by the HDR synchronization technology described in this disclosure. As will be explained in more detail below, camera processor 14 may include and / or be configured to execute HDR synchronizer 19. That is, HDR synchronizer 19 may be software executed by camera processor 14, firmware executed by camera processor 14, or dedicated hardware within camera controller 14. Figure 1 The HDR synchronizer 19 is shown to be separate from the ISP 23, but in some examples, the HDR synchronizer 19 may be part of the ISP 23.

[0052] HDR synchronizer 19 can be configured to determine the FOV and resolution of the image output by the primary camera (e.g., image sensor 12A of camera 15). Camera processor 14 can process the image output by image sensor 12A to generate an HDR image. HDR synchronizer 19 can instruct different secondary cameras (e.g., image sensor 12B of camera 15) to synchronize both the FOV and resolution of the images output by the secondary cameras to be the same as those output by the primary camera (e.g., sensor 12A of camera 15). Camera processor 14 can also generate HDR images from the output of the secondary cameras. For example, camera processor 14 (e.g., ISP 23) can also perform tone alignment on the image output by the secondary camera (e.g., image sensor 12B of camera 15) to match the tonal range (e.g., range of brightness levels and / or colors) presented in the HDR image generated from the video output by the primary camera (e.g., sensor 12A of camera 15).

[0053] In one example, camera processor 14 can achieve tone alignment by performing the same HDR processing on the image received from the secondary camera as it does on the image received from the primary camera. In another example, camera processor 14 can perform tone alignment on the image received from the secondary camera using a different HDR technique, where the different HDL technique approximates the tonal range produced by the HDR method used on the image output from the primary camera. In this way, camera processor 14 can use two HDR images synchronized in terms of FOV, resolution, and tone to calculate a depth map, thereby improving the accuracy of the depth map. This more accurate depth map can then be used to improve the quality of any image effects (including defocusing) applied to the image (including the HDR image).

[0054] Accordingly, as will be described in more detail below, in one example of this disclosure, camera processor 14 may be configured to receive one or more first images from a first image sensor (e.g., image sensor 12A), wherein the one or more first images have a first field of view (FOV) and a first resolution. Camera processor 14 may also receive one or more second images from a second image sensor (e.g., image sensor 12B), wherein the one or more second images have a second FOV and a second resolution. Camera processor 14 may also generate a first high dynamic range (HDR) image based on the one or more first images.

[0055] Camera processor 14 can perform tone alignment on one or more second images based on a first HDR image to produce a second HDR image. In some examples, such as when the first image sensor and the second image sensor have different native FOVs (e.g., the first image sensor is a wide-angle sensor and the second image sensor is an ultra-wide-angle sensor), camera processor 14 can also be configured to synchronize the second FOV of one or more second images with the first FOV of one or more first images, and to synchronize the second resolution of one or more second images with the first resolution of one or more first images. In this example, camera processor 14 performs tone alignment on one or more second images after synchronizing the second FOV and the second resolution.

[0056] Camera processor 14 can then use the first HDR image and the second HDR image to generate a depth map, and can then use the depth map to apply image effects to the first HDR. In one example, the image effect is a defocusing effect. Camera processor 14 can also be configured to apply other image effects to the first HDR image using the calculated depth map. Other image effects may include filtering, masking, customizable expressions, green screen effects, background replacement effects, semantic understanding, 3D pose estimation and / or tracking, depth-assisted autofocus, flash measurement, and / or any other image effects and / or image processing techniques that can utilize the depth map determined from the HDR image.

[0057] Despite the various structures of computing device 10 Figure 1 While illustrated separately, the technology disclosed herein is not limited thereto, and in some examples, these structures can be combined to form a System-on-a-Chip (SoC). As an example, the camera processor 14, CPU 16, GPU 18, and display interface 26 can be formed on a common integrated circuit (IC) chip. In some examples, one or more of the camera processor 14, CPU 16, GPU 18, and display interface 26 can be formed on separate IC chips. Furthermore, in some examples, the HDR synchronizer 19 of the camera processor 14 can be part of an ISP 23. Various other permutations and combinations are possible, and the technology disclosed herein should not be considered limited to... Figure 1 The example shown. In one example, CPU 16 may include camera processor 14, such that one or more camera processors 14 are part of CPU 16. In such an example, CPU 16 may be configured to perform one or more of the various techniques otherwise attributed herein to camera processor 14. For the purposes of this disclosure, camera processor 14 will be described herein as separate from and distinct from CPU 16, although this may not always be the case.

[0058] Figure 1The various structures illustrated in the diagram can be configured to communicate with each other using bus 32. Bus 32 can be any of different bus architectures, such as a third-generation bus (e.g., HyperTransport bus or InfiniBand bus), a second-generation bus (e.g., Advanced Graphics Port bus, Peripheral Component Interconnect (PCI) Express bus, or Advanced Extensible Interface (AXI) bus), or another type of bus or device interconnect. It should be noted that... Figure 1 The specific configurations of the bus and communication interfaces between the different structures shown are merely exemplary, and other configurations of computing devices and / or other image processing systems with the same or different structures can be used to implement the techniques disclosed herein.

[0059] also, Figure 1 The various components shown in the figure (whether formed on one device or on different devices), including sensor 12 and camera processor 14, can be formed as at least one or a combination of fixed-function or programmable circuitry, such as integrated or discrete logic circuitry in one or more microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or other equivalents. Furthermore, examples of local memory 20 include one or more volatile or non-volatile memories or storage devices, such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media, or optical storage media.

[0060] In some examples, memory controller 24 can facilitate the forwarding of data in and out of system memory 30. For example, memory controller 24 can receive memory read and write commands and maintain such commands relative to memory 30 to provide storage services to various components of computing device 10. In such examples, memory controller 24 can be communicatively coupled to system memory 30. Although memory controller 24 in Figure 1 In some examples, the computing device 10 is illustrated as a separate processing circuit from both the CPU 16 and the system memory 30. However, in some examples, some or all of the functions of the memory controller 24 may be implemented on one or more of the CPU 16, system memory 30, camera processor 14, video encoder / decoder 17, and / or GPU 18.

[0061] System memory 30 may store program modules and / or instructions and / or data accessible by camera processor 14, CPU 16, and / or GPU 18. For example, system memory 30 may store user applications (e.g., instructions for a camera application), composite images from camera processor 14, etc. System memory 30 may additionally store information used by and / or generated by other components of computing device 10. For example, system memory 30 may act as device memory for camera processor 14. System memory 30 may include one or more volatile or non-volatile memories or storage devices, such as RAM, SRAM, DRAM, ROM, EPROM, EEPROM, flash memory, magnetic data media, or optical storage media. Furthermore, system memory 30 may store image data (e.g., video data frames, encoded video data, sensor mode settings, zoom settings, 3A parameters, etc.). In some examples, system memory 20 or local memory 30 may store image data in on-chip memory, such as in a memory buffer of system memory 30 or local memory 20. In another example, system memory 30 or local memory 20 can output image data for storage outside the memory of the chip or buffer, such as storage in a secure digital storage device (SD card). TM The system memory 30 or local memory 20 may be stored on a card or, in some cases, in another internal memory of the camera device. In an illustrative example, the system memory 30 or local memory 20 may be implemented as a buffer memory on the camera processor 14 chip, the GPU 18 chip, or both (where a single chip includes two processing circuits).

[0062] In some examples, system memory 30 may include instructions that cause camera processor 14, CPU 16, GPU 18, and / or display interface 26 to perform the functions attributed to these components in this disclosure. Therefore, system memory 30 may be a computer-readable storage medium storing thereon instructions that, when executed, cause one or more processors (e.g., camera processor 14, CPU 16, GPU 18, and display interface 26) to perform various techniques of this disclosure.

[0063] In some examples, system memory 30 is a non-transitory storage medium. The term "non-transitory" means that the storage medium is not implemented over a carrier wave or propagating signal. However, the term "non-transitory" should not be construed as meaning that system memory 30 is immovable or that its contents are static. As an example, system memory 30 can be removed from computing device 10 and moved to another device. As another example, a memory substantially similar to system memory 30 can be inserted into computing device 10. In some examples, non-volatile storage media are capable of storing data that changes over time (e.g., in RAM).

[0064] Furthermore, the camera processor 14, CPU 16, and GPU 18 can store image data, user interface data, etc., in corresponding buffers allocated within system memory 30. Display interface 26 can retrieve data from system memory 30 and configure display 28 to display an image represented by the image data (such as via the screen of user interface 22). In some examples, display interface 26 may include a digital-to-analog converter (DAC) configured to convert digital values ​​retrieved from system memory 30 into analog signals that can be consumed by display 28. In other examples, display interface 26 can directly pass these digital values ​​to display 28 for processing.

[0065] Computing device 10 may include a video encoder and / or a video decoder 17, either of which may be integrated as part of a combined video encoder / decoder (CODEC) (e.g., a video codec). The video encoder / decoder 17 may include a video encoder that encodes video captured by one or more cameras 15 or a decoder capable of decoding compressed or encoded video data. In some cases, CPU 16 and / or camera processor 14 may be configured to encode and / or decode video data, in which case CPU 16 and / or camera processor 14 may include the video encoder / decoder 17.

[0066] CPU 16 may include a general-purpose or special-purpose processor that controls the operation of computing device 10. A user may provide input to computing device 10 to cause CPU 16 to execute one or more software applications. These software applications executing on CPU 16 may include, for example, camera applications, graphics editing applications, media player applications, video game applications, graphical user interface applications, or other programs. For example, a camera application may allow a user to control various settings of camera 15. A user may provide input to computing device 10 via one or more input devices (not shown) (such as a keyboard, mouse, microphone, touchpad, or another input device coupled to computing device 10 via user interface 22).

[0067] An example software application is a camera application. The CPU 16 executes the camera application, and in response, the camera application causes the CPU 16 to generate the content output by the display 28. For example, the display 28 may output information such as light intensity, whether the flash is enabled, and other such information. The camera application may also cause the CPU 16 to instruct the camera processor 14 to process the image output by the sensor 12 in a user-defined manner. A user of the computing device 10 can interface with the display 28 (e.g., via user interface 22) to configure how the image is generated (e.g., using zoom settings, using or not using a flash, focus settings, exposure settings, video or still images, and other parameters). For example, the CPU 16 may receive instructions via user interface 22 for capturing an image in HDR mode (e.g., capturing and generating an HDR image) and / or for applying depth-of-field effects (such as defocus effects) (e.g., in a so-called "portrait" mode). The portrait mode may also include defocus settings, which indicate the amount of blur to be applied.

[0068] Display 28 may include a monitor, television, projection device, HDR display, liquid crystal display (LCD), plasma display panel, light-emitting diode (LED) array, organic LED (OLED), electronic paper, surface conduction electron emission display (SED), laser television display, nanocrystal display, or other types of display units. Display 28 may be integrated within computing device 10. For example, display 28 may be the screen of a mobile phone, tablet, or laptop. Alternatively, display 28 may be a standalone device coupled to computing device 10 via a wired or wireless communication link. For example, display 28 may be a computer monitor or flat panel display connected to a personal computer via a cable or wireless link. Display 28 may provide a preview frame that a user can view to see what is being stored or what images might be seen when camera 15 is used to actually take a picture or begin recording video.

[0069] In some examples, camera processor 14 may output frame streams to memory controller 24 for storage as video files. In some examples, CPU 16, video encoder / decoder 17, and / or camera processor 14 may output HDR images (e.g., with or without defocus) for storage as video files. In some examples, memory controller 24 may generate and / or store output frames in any suitable video file format. In some examples, video encoder / decoder 17 may encode output frames before CPU 16, video encoder / decoder 17, and / or camera processor 14, such that these output frames are stored as encoded video. Encoder / decoder 17 may encode image data frames using various encoding techniques, including those described by standards and extensions defined by MPEG2, MPEG4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High-Efficiency Video Coding (HEVC), Universal Video Coding (VCC), etc. In a non-limiting example, the CPU 16, video encoder / decoder 17, and / or camera processor 14 may enable the output frames to be stored using the Moving Picture Experts Group (MPEG) video file format.

[0070] Figure 2 A more detailed map is shown. Figure 1 A block diagram of example components of a computing device. (See attached diagram.) Figure 2 As shown, the main camera 15A (e.g., including image sensor 12A) can be instructed by the camera processor 14 to output one or more images for generating an HDR image. The secondary camera 15B (e.g., including image sensor 12B) can be instructed by the camera processor 14 to output one or more images that can be used to calculate a depth map, which can be used to apply one or more image effects (e.g., defocusing effect) based on image segmentation to the HDR image generated from the output of the main camera 15A.

[0071] exist Figure 2In the example, the ISP 23 of the camera processor 14 includes an HDR generator 44, a tone alignment unit 46, a depth map calculator 21, and an image effects applicator 48. Each of the HDR generator 44, tone alignment unit 46, depth map calculator 21, and image effects applicator 48 can be implemented as software, firmware, hardware, or any combination thereof. The main camera 15A can output one or more master images to the HDR generator 44. The HDR generator 44 can use the one or more master images to create a first HDR image. In one example, the one or more master images are multiple images, each captured using a different exposure setting. The HDR generator 44 can be configured to blend the multiple images together, resulting in a wide range of brightness values ​​in the blended image (i.e., the image has a high dynamic range).

[0072] For example, camera processor 14 can execute an automatic exposure algorithm that determines two or more different exposure settings for image sensor 12. For three image examples, camera processor 14 can determine a short exposure to preserve detail in the bright areas of the scene, a long exposure to enhance detail in the dark areas of the scene, and a medium-length exposure to preserve more precise brightness levels of the midtones in the scene. Camera processor 14 can then fuse the images generated from the three different exposures to produce an HDR image.

[0073] The aforementioned technique used for HDR generation can be referred to as multi-frame HDR (MFHDR). The HDR generator 44 can perform MFHDR on the main image output by the main camera 15A using software or hardware techniques. MFHDR is not the only technique that the HDR generator 44 can use to generate HDR images. HDR techniques can include both multi-frame HDR and single-frame HDR. Multi-frame HDR typically uses a combination of images captured at different exposures. In some examples, such as the aforementioned MFHDR, HDR techniques can be applied as post-processing by the camera processor 14 (e.g., ISP 23). In other examples, HDR techniques can be applied "on the sensor." That is, the main camera 15A can perform processing such that the image output by the main camera 15A is already an HDR image. One example of such processing is called 3-exposure HDR. In 3-exposure HDR, the main camera 15A can output an image comprising pixels captured at short, medium, and long exposure settings, and then separate that image into three images (e.g., a short exposure image, a medium exposure image, and a long exposure image). The main camera 15A can then blend the three resulting images together before sending them to the camera processor 14.

[0074] In other examples, the HDR generator 44 can generate an HDR image (e.g., a first HDR image) using a single frame output from the main camera 15A. Single-frame HDR can be post-processed using histogram equalization on the individual image, a technique sometimes referred to as Local Tone Mapping (LTM). In other examples, the HDR generator 44 can generate an HDR image from the output of the main camera 15A using Adaptive Dynamic Range Coding (ADRC). When performing ADRC, the camera processor 14 can preserve the bright areas of the image by capturing it with a lower sensor exposure. The camera processor can then apply digital gain to the output image to compensate for the overall brightness.

[0075] Another example technique for generating HDR images is interleaved HDR. In interleaved HDR, camera processor 14 can be configured to cause camera 15 to use multiple rolling shutters to capture multiple images. Instead of generating an image from the entire scene at once, the rolling shutters generate images by scanning rows or columns of image sensor 12. For example, camera 15 can combine long-period rolling shutter captures with short-period rolling shutter captures. Camera 15 can also perform row interleaving on the outputs of the two rolling shutters.

[0076] According to the technology disclosed herein, the HDR synchronizer 19 can be configured to determine the resolution of the image output by the main camera 15A, the field of view (FOV) of the image output by the camera 15A, and the HDR technique used by the HDR generator 44 and / or the main camera 15A (e.g., for "on-sensor" HDR) when generating the first HDR image. That is, the HDR synchronizer 19 can determine the FOV and resolution of the main sensor 90 of the main camera 15A and the HDR technique 94 of the HDR generator 44. Based on this information (i.e., the FOV and resolution of the main sensor 90 and the HDR technique 94), the HDR synchronizer 19 will send FOV and resolution synchronization 92 to the secondary camera 15B and tone alignment 96 to the tone alignment unit 46.

[0077] The FOV and resolution synchronization 92 include instructions to cause the secondary camera 15B to output a sub-image with the same FOV as the main camera 15A and the same FOV as the secondary camera 15B. For example, the main camera 15A may include a so-called "wide-angle" camera lens. To use a defocus effect, the camera processor 14 may instruct the main camera 15A to output an FOV equivalent to that of a camera with a telephoto lens. Generally, a telephoto lens on a mobile device is approximately twice the zoom level of a wide-angle lens. For example, if the main camera 15A is a wide-angle lens, the camera processor 14 may instruct the main camera 15A to output one-quarter of the full FOV of the wide-angle lens. This is because one-quarter of the full FOV of a wide-angle lens can be equivalent to the full FOV of a telephoto lens. However, any size FOV can be used.

[0078] In this example, the secondary camera 15B may be a camera that outputs a different FOV than the primary camera 15A. Many mobile devices now include two, three, four, or more cameras, each capable of outputting images with a different FOV. In one example, the secondary camera 15B may include a so-called “ultra-wide” camera lens, which has less zoom and a larger FOV than a wide-angle lens. Preferably, the primary camera 15A and the secondary camera 15B are physically positioned in the same vertical or horizontal plane (e.g., coplanar) on the computing device 10 and are not physically oriented diagonally. While possible, images from two cameras diagonally positioned relative to each other would make it difficult to calculate depth values ​​from such cameras.

[0079] In some example mobile devices, cameras with ultra-wide-angle and wide-angle lenses are coplanar (e.g., on the same vertical or horizontal plane), while a camera with a telephoto lens is located diagonally opposite the ultra-wide-angle and wide-angle lenses. In such devices, when calculating depth values ​​for depth-of-field effects (such as defocusing), it would be preferable to use two coplanar camera sensors as a primary camera 15A and a secondary camera 15B. For example, if the primary camera 15A has a wide-angle lens and is outputting a FOV that is one-quarter of its full FOV, and the secondary camera 15A has an ultra-wide-angle lens, then the HDR synchronizer 19 can send the FOV and resolution synchronization 92 to the secondary camera 15B, which instructs the secondary camera 15B to output an FOV that matches the one-quarter FOV of the primary camera 15A.

[0080] For some exemplary ultra-wide-angle lenses, the output FOV of the secondary camera 15B can be one-sixteenth of the full FOV of the secondary camera 15B. Generally, the primary camera 15A and the secondary camera 15B can be any two types of cameras. The two cameras can output the same full FOV (e.g., the camera's maximum FOV), or they can be two cameras capable of outputting different full FOVs. Regardless of the FOVs that can be output by the primary camera 15A and the secondary camera 15B, the HDR synchronizer 19 sends FOV resolution synchronization 92 to the secondary camera 15B such that the output FOV of the secondary image is the same as the FOV of the primary image output by the primary camera 15A. In other examples, the camera processor 14 will receive the full FOV of the secondary camera 15B and perform arbitrary FOV synchronization at the ISP 23, CPU 16, or another processing unit of the computing device 10, rather than instructing the secondary camera 15B to perform FOV synchronization.

[0081] As described above, FOV and resolution synchronization 92 may also include instructions to cause the secondary camera 15B to output an image at the same resolution as the primary camera 15A. In this context, resolution refers to the number of pixels output in an image. Resolution is typically measured against the camera's full FOV, and may be even smaller if the camera's output FOV is smaller than the full FOV. The primary camera 15A and this camera 15B may include image sensors capable of outputting images at multiple different resolutions. Generally, the HDR synchronizer 19 may determine the output resolution of the primary camera 15A and include instructions regarding FOV and resolution synchronization 92 that cause the secondary camera 15B to output an image at the same resolution as the primary camera 15A.

[0082] As an example, the main camera 15A may include a Quad Bayer Filter Array (QCFA) sensor capable of outputting images at 48MP (or any other resolution) in remosaic mode and at 12MP (e.g., a four-fold reduction from the maximum resolution) in binning mode. In other examples, other levels of binning may be used. Generally, in binning mode, the sensor averages adjacent pixels (e.g., two or four adjacent pixels) with the same filter to provide a higher signal-to-noise ratio for capturing images in lower light conditions. Further details regarding binning and remosaic will be described below.

[0083] For the example of a QCFA sensor, the HDR synchronizer 19 can determine whether the primary camera 15A outputs an image using a re-mosaic resolution mode or a merge readout resolution mode (e.g., in the FOV and resolution of the primary sensor 90). Based on this determination, the HDR synchronizer 19 will include instructions regarding FOV and resolution synchronization 92, which cause the secondary camera 15B to output an image using the same resolution mode. In some examples, if the FOV on the primary camera 15A or the secondary camera 15B is a small portion of the full FOV (e.g., for a wide-angle or ultra-wide-angle lens), the primary camera 15A and / or the secondary camera 15B can magnify the output image so that the same number of pixels exist in both the output of the primary camera 15A and the output of the secondary camera 15B.

[0084] The result of FOV and resolution synchronization 92 is that the secondary camera 15B outputs an image with FOV and resolution matching that of the primary camera 15A. In this way, the secondary image output by the secondary camera 15B matches the primary image output by the primary camera 15A more closely. Thus, corresponding pixels in each of the primary and secondary images may exhibit the same features in the output image, thereby causing the depth map calculator 21 to more accurately determine the depth values ​​of the features represented by these pixels.

[0085] Furthermore, to match the FOV and resolution of the primary image, the HDR synchronizer 19 can also be configured to generate an HDR image from the output of the secondary camera 15B. For example, the HDR synchronizer 19 can be configured to match the hue of the secondary image with the hue of a first HDR image created by the HDR generator 44 from the primary image. Generally, compared to an image with a standard or low dynamic range, the HDR image will include a higher range of luminance values ​​(e.g., higher dynamic range luminance) and / or a higher range of color values ​​(e.g., a wider color gamut). To match the hue range of the first HDR image to improve depth map calculation, the HDR synchronizer 19 can determine the HDR technique 94 used by the HDR generator 44 to generate the first HDR data and send a hue alignment 96 to the hue alignment unit 46. The hue alignment unit 46 will use the hue alignment 96 instruction to perform HDR processing on the secondary image to produce a second HDR image, such that the second HDR image includes the same or approximately the same hue range as the first HDR image generated by the HDR generator 44.

[0086] In some examples, tone alignment 96 may include instructions for tone alignment 46 to perform the same HDR processing as HDR generator 44. For example, if HDR synchronizer 19 determines that the HDR technique 94 of HDR generator 44 is hardware-based MFHDR, then HDR synchronizer 19 will include instructions regarding tone alignment 96 that cause tone alignment 46 to also perform hardware-based MFHDR on the secondary image output by secondary camera 15B. Similarly, if HDR synchronizer 19 determines that the HDR technique 94 of HDR generator 44 is ADRC, then HDR synchronization device 19 will include instructions regarding tone alignment 96 that cause tone alignment 46 to also perform ADRC on the secondary image output by secondary camera 15B.

[0087] However, in some examples, computing device 10 may not be configured to perform the same HDR processing on multiple sets of images simultaneously. For example, computing device 10 may include only one processing engine in ISP 23 capable of performing MFHDR. In this case, HDR synchronizer 19 may include instructions regarding tone alignment 96, which cause tone aligner 46 to apply different HDR processing, such as ADRC, 3-exposure HDR, interleaved HDR, Quad Bayer coded (QBC) HDR, or any other HDR technique. In other examples, HDR synchronizer 19 may include instructions regarding tone alignment 96, which are then sent to secondary camera 15B, causing secondary camera 15A to perform "on-sensor" HDR techniques, such as 3-exposure HDR.

[0088] In an example where the same HDR processing used by HDR generator 44 is unavailable for images output by secondary camera 15B, and ISP 23 can have multiple different HDR techniques for tone alignment 46 to use, HDR synchronizer 19 can determine which of the available HDR processing techniques most closely approximates the tonal range of the HDR image produced by the processing performed by HDR generator 44. HDR synchronizer 19 can then instruct tone alignment 46 to use the determined HDR technique to generate a second HDR image that most closely approximates the tonal range produced by HDR generator 44.

[0089] Using the techniques disclosed herein, the second HDR image output by tone alignment 46 matches and / or closely approximates the FOV, resolution, and tone of the first HDR image. The depth map calculator can then determine the relative depth of pixels in the first HDR image by comparing the first and second HDR images. Because the first and second HDR images were captured from slightly different perspectives (i.e., the main camera 15A and the secondary camera 15B are physically separated), the differences in position, orientation, and feature size represented by pixels in the two images are used by the depth map calculator to calculate the depth of each pixel (e.g., a depth map). The depth map includes the distance of each pixel in the image relative to the image sensor, or the distance of each pixel in the image relative to the focal point.

[0090] The depth map calculator 21 can then send the calculated depth map to the image effects applicator 48. In some examples, in addition to determining the depth map from the first HDR image and the second HDR image (e.g., from the stereo HDR image) as described above, the depth map calculator 21 can perform additional techniques to refine and / or determine the depth values ​​of features in the first HDR. Other techniques for determining and / or refining depth values ​​can include artificial intelligence (AI) segmentation-assisted stereo depth (DFS), time-of-flight (ToF)-assisted DFS, structured light-assisted DFS, etc. For example, the depth map calculator can use a ToF laser to measure the distance to features in the first HDR image. The depth map calculator 21 can use this distance to perform fine timing on the depth maps generated from the first HDR image and the second HDR image.

[0091] Image effect applicator 48 can use this depth map to apply depth-of-field effects (such as defocusing) to the first HDR image generated from the output of the main camera 15A. For example, when applying a defocusing effect, image effect applicator 48 can use the calculated depth map to gradually blur pixels in the first HDR image. For example, image effect applicator 48 can keep pixels at a depth matching the focus distance of the main image in focus (e.g., without applying blur), while image effect applicator 48 can gradually blur pixels at a greater focus distance. The farther a pixel is from the focus distance, the more blur image effect applicator 48 can apply to achieve the defocusing effect.

[0092] Figure 3 This is a conceptual diagram illustrating examples of FOV synchronization for different lens types. (Example:) Figure 3As shown, in some examples, image 320 captured using a camera with a telephoto lens has approximately a 2x zoom FOV (e.g., telephoto 2x FOV) compared to image 310 captured using a camera with a wide-angle lens (e.g., wide-angle FOV). That is, in this example, the wide-angle lens and wide-angle FOV can be considered as 1x zoom, while the telephoto lens and telephoto FOV can be considered as 2x zoom. Thus, as shown by the dashed lines, if the FOV of image 310 matches the full FOV of image 320 (e.g., telephoto 2x FOV), the output FOV from image 310 will be one-quarter of the full wide-angle FOV of image 310. As mentioned above, in some examples, the defocus effect is most aesthetically pleasing when captured with the FOV of a telephoto lens. Figure 3 As shown, the camera (e.g., Figure 2 The main camera (15A) can crop the wide-angle FOV of the image 310 to produce an output FOV equal to the full FOV of a 2x zoom telephoto lens.

[0093] For cameras with ultra-wide-angle lenses (e.g., producing an ultra-wide-angle FOV) (e.g., Figure 2 In the case of the secondary camera 15B, to output the same FOV as the telephoto 2x telephoto lens, the camera will crop image 300 so that one-sixteenth of the full FOV of image 300 is output. That is, in some examples, one-sixteenth of the FOV of the ultra-wide-angle camera is the same as the FOV of the 2x zoom telephoto lens. The output FOV of the ultra-wide-angle lens matching the full FOV of image 310 will be one-quarter the size of image 300.

[0094] Figure 4 This is a block diagram illustrating example field-of-view synchronization techniques using wide-angle and ultra-wide-angle FOV. (Example:) Figure 4 As shown, the main sensor 12A is part of a camera that includes a wide-angle lens capable of capturing a wide-angle FOV 400. The secondary sensor 12B is part of a camera that includes an ultra-wide-angle lens capable of capturing an ultra-wide FOV 410. The camera processor 14 can be configured to instruct the main sensor 12A to output a wide-angle 2x FOV 430. That is, the camera processor 14 can be configured to instruct the main sensor 12A, for example, when generating an image that will apply a defocusing effect, to output an FOV equivalent to that of a telephoto lens zoomed 2x relative to the full-width FOV. Figure 4 As shown, the main sensor 12A can be configured to crop the center quarter of the full wide-angle FOV 400 to output a wide-angle 2x FOV 430.

[0095] The HDR synchronizer 19 can be configured to determine the output FOV of the primary sensor 12B directly from the camera processor 14 or by querying the primary sensor 12A. Based on the output FOV of the primary sensor 12A and the sensor and lens type of the secondary sensor 12B, the HDR synchronizer 19 sends the FOV and resolution synchronization 92 to the secondary sensor 12B. The FOV and resolution synchronization 92 will include instructions that cause the secondary sensor 12B to output an FOV (e.g., Wide 2x FOV 440) that matches the output Wide 2x FOV 430 of the primary sensor 12A.

[0096] exist Figure 4 In the example, the secondary sensor 12B is part of a camera that includes an ultra-wide-angle lens capable of capturing an ultra-wide-angle FOV 410. To match the output Wide 2x FOV 430 of the primary sensor 12A, the FOV and resolution synchronization 92 can instruct the secondary sensor 12B to crop the ultra-wide-angle FOV 410 such that one-sixteenth of the ultra-wide-angle FOV 410 is output to achieve an output Wide 2x FOV 440. Or more generally, the FOV and resolution synchronization 92 can instruct the secondary sensor 12B to crop the ultra-wide-angle FOV 410 such that the output FOV of the secondary sensor 12B matches the output FOV of the primary sensor 12A.

[0097] The primary sensor 12A and secondary sensor 12B can achieve a specific output FOV by cropping a portion of the initial image to output the full FOV. However, to match the output FOV of the primary sensor 12A, the secondary sensor 12B does not necessarily need to be cropped to the center of the full ultra-wide-angle FOV 410. In some examples, the matching wide-angle FOV in the ultra-wide-angle FOV 410 may not be centered. Thus, in some examples, the FOV and resolution synchronization 92 may further include instructions for spatially aligning the FOV output by the secondary sensor 12B.

[0098] After any cropping and spatial alignment, the primary sensor 12A and secondary sensor 12B can send an image with an output FOV to the ISP 23 for ISP processing. As described above, the ISP processing may include 3A statistical processing, as well as an HDR generator 44, a tone alignment unit 46, a depth map calculator 21, and an image effects applicator 48, such as... Figure 2 As shown. Figure 4 The HDR synchronizer 19 is shown as separate from the ISP 23. However, in some examples, the HDR synchronizer 19 may be part of the ISP 23.

[0099] In other examples, instead of instructing the primary sensor 12A and secondary sensor 12B to perform any cropping to achieve FOV synchronization, the camera processor 14 can be configured to receive the full FOV of both the primary sensor 12A and secondary sensor 12B. The HDR synchronizer 19 can then enable the ISP 23, CPU 16, or another processing unit of the computing device 10 to perform FOV synchronization.

[0100] Figure 5 The figure illustrates an example field-of-view and resolution synchronization technique of this disclosure. Figure 5 In the example, HDR synchronizer 19 determines that the main sensor 12A of the main camera 15A is configured to output an image using a merged readout resolution mode and is configured to crop the field of view (FOV) to a 2x zoom. That is, the output FOV of the main sensor 12A is equivalent to the output FOV of a telephoto lens relative to a wide-angle lens at 2x zoom. HDR synchronizer 19 can then send the FOV and resolution synchronization 92 to the secondary sensor 12B of the secondary camera 15B. The FOV and resolution synchronization 92 includes instructions to cause the secondary sensor 12B to output an image using the merged readout resolution mode and to cause the secondary sensor to crop the image to an FOV equivalent to a 2x zoom.

[0101] Figure 6 An example resolution synchronization technique of this disclosure is shown. Figure 6 A portion of the QCFA sensor section 602A is shown. The QCFA 602A features a Quad Bayer mosaic pattern. In one example, each 16x16 section of the QCFA sensor may include four adjacent red filters, four adjacent blue filters, and two sets of four adjacent green filters. By combining them, such a pattern can represent a wide range of colors. More green filters are used because the human eye is more sensitive to green.

[0102] A QCFA sensor includes four adjacent color pixels because the QCFA sensor is configured for a merge readout mode. In merge readout mode, adjacent color pixels are averaged together to form a single output pixel. Averaging the values ​​of four adjacent color pixels increases the signal-to-noise ratio of the output, especially in low-light scenes, at the cost of resolution. For example, when using 4:1 merge readout, a 48MP QCFA sensor will output an image at 12MP. QCFA sensor section 602C illustrates the output of QCFA sensor section 602A after 4:1 merge readout. Although Figure 6 The example illustrates a 4:1 merge readout, but the techniques disclosed herein can be used for any level of merge readout.

[0103] For example, as described in U.S. Patent Application No. 16 / 667662, entitled “IMAGE CAPTURE MODEADAPTATION”, filed October 29, 2019 by Liu et al., in some zoom operations, the camera processor 14 may use pixel-binning readout techniques to adapt to increasing or decreasing zoom levels. More generally, the image sensor 12 may perform pixel-binning readout (e.g., 4x4, 3x3, 2x2, horizontal, vertical, etc.) by combining multiple pixels of the image sensor 12 into fewer pixels for output to the camera processor 14. The pixel-binning readout technique improves the signal-to-noise ratio (SNR) by enabling the image sensor 12 (or, in some instances, the camera processor 14) to combine pixels through various combination schemes, including averaging or summing multiple pixels for each output pixel.

[0104] In other examples, the QCFA sensor can be configured to output full resolution, such as in scenes where readout merging is not required (e.g., scenes with relatively high highlights). Because the pattern of the QCFA sensor is arranged for readout merging purposes, the QCFA sensor can be configured to perform remosaicing on the output color pixels, resulting in a traditional Bayer pattern. QCFA sensor section 602B illustrates QCFA sensor section 602A after remosaicing. As can be seen, the remosaicing process preserves the sensor's full resolution.

[0105] In the re-mosaic example Figure 6 A QCFA sensor section 602A in a Bayer mosaic state is shown as being re-mosaiced into a QCFA sensor section 602B. Re-mosaicing involves converting a first mosaic into a second mosaic. For this purpose, a processing system (e.g., a processor of image sensor 12 and / or camera processor 14) can re-mosaic based on the context of the first mosaic. The processing system can assign a context to the first mosaic based on one or more statistical measures of the first mosaic, such as variance and / or standard deviation. A first mosaic with a complex structure (e.g., many edges) can produce a high variance and / or standard deviation, resulting in a corresponding first context. Conversely, a first mosaic with less structure (e.g., few edges) can produce a low variance and / or standard deviation, resulting in a corresponding second context. The first mosaic can have a first spectral pattern (e.g., a Quadra pattern). The second mosaic can have a second spectral pattern (e.g., a Bayer pattern). In some examples, the second spectral pattern can be compatible with a multichannel interpolator.

[0106] In some examples, the first and second mosaics can have the same resolution (and therefore the same pixels arranged in the same aspect ratio). In other examples, the second mosaic can have a different resolution than the first mosaic. In one example, the second mosaic can have a lower resolution than the first mosaic and therefore include some, but not all, of the pixels in the first mosaic.

[0107] In some cases, image sensor 12 can output all pixels of its full potential field of view (FOV) (e.g., 100% of the FOV). However, in other cases, camera processor 14 can cause image sensor 12 to output a reduced number of pixels relative to its full potential FOV (e.g., 33% in the center, 25% in the center, etc.). In other words, the term "output pixels" as used herein generally refers to the actual number of pixels output from image sensor 12 to camera processor 14 (e.g., 100%, 25% in the center, etc.).

[0108] Figure 7 The figure illustrates another example field-of-view and resolution synchronization technique for a Quad Bayer color filter array (QCFA) according to this disclosure. In this example, both the primary sensor 12A and the secondary sensor 12B are QCFA sensors. Thus, the HDR synchronizer 19 can determine whether the primary sensor 12A outputs an image in a re-mosaic mode 700A or a merge readout mode 710A. Based on this determination, the HDR synchronizer 19 will instruct the secondary sensor 12B to output an image in either merge readout mode 710B or re-mosaic mode 700B to match the output of the primary sensor 12A. Figure 7 In the example, both the primary sensor 12A and the secondary sensor 12B are configured to be cropped to a 2x zoom FOV. In a merge readout context, the sensors can also perform magnification to match the FOV, since the sensor resolution is reduced during merge readout. However, the techniques disclosed herein can use FOVs other than 2x zoom.

[0109] After synchronizing resolution and FOV, the main sensor 12A and the secondary sensor 12B can send an image with output FOV and resolution to the ISP 23 for ISP processing. As described above, the ISP processing may include 3A statistical processing, as well as an HDR generator 44, a tone alignment unit 46, a depth map calculator 21, and an image effects applicator 48, such as... Figure 2 As shown. Figure 7 The HDR synchronizer 19 is shown as separate from the ISP 23. However, in some examples, the HDR synchronizer 19 may be part of the ISP 23.

[0110] Figure 8 This is a flowchart illustrating an example operation of camera processing according to the example technology of this disclosure. Figure 8 The technology can be generated by one or more processors (such as...) Figure 1 The camera processor 14 is configured to perform this operation. In one example of this disclosure, the camera processor 14 may be configured to receive one or more first images from a first image sensor, wherein the one or more first images have a first field of view (FOV) and a first resolution (800). The camera processor 14 may also generate a first high dynamic range (HDR) image (802) from the one or more first images.

[0111] The camera processor 14 can also generate a second HDR image (804). For example, the camera processor 14 can be further configured to receive one or more second images from a second image sensor, wherein the one or more second images have a second FOV and a second resolution. (Refer to below) Figure 9 As explained, camera processor 14 can be configured to generate a second HDR image from one or more second images. For example, the camera processor can synchronize the second FOV and second resolution of one or more second images with the first FOV and first resolution of one or more first images. Furthermore, camera processor 14 can apply the same HDR processing applied to one or more second images as applied to one or more first images to produce a first HDR image.

[0112] Camera processor 14 can then use the first HDR image and the second HDR image to generate a depth map (806), and use the depth map to apply an image effect to the first HDR image (808). In one example, the image effect is a defocusing effect. As explained above, in some examples, camera processor 14 can also be configured to synchronize both the FOV and resolution output by the second image sensor to be the same as the FOV output and resolution of the first image sensor. Figure 9 The processing, including FOV and resolution synchronization, is described in more detail.

[0113] Figure 9 This is a flowchart illustrating another example operation of camera processing according to the example technology of this disclosure. Figure 9 The technology can be generated by one or more processors (such as...) Figure 1 The camera processor 14) is used to execute this.

[0114] In one example of this disclosure, camera processor 14 may be configured to receive one or more first images from a first image sensor, wherein the one or more first images have a first field of view (FOV) and a first resolution (900). Camera processor 14 may be further configured to receive one or more second images from a second image sensor, wherein the one or more second images have a second FOV and a second resolution (902). Camera processor 14 may be further configured to synchronize the second FOV of the one or more second images with the first FOV of the one or more first images (904), and to synchronize the second resolution of the one or more second images with the first resolution of the one or more first images (906).

[0115] Camera processor 14 can generate a first high dynamic range (HDR) image (908) from one or more first images, and can perform tone alignment on one or more second images based on the first HDR image to produce a second HDR image (910). Camera processor 14 can generate a depth map (912) using the first HDR image and the second HDR image, and can apply an image effect to the first HDR using the depth map (914). In one example, the image effect is a defocusing effect.

[0116] In one example of this disclosure, in order to synchronize a second FOV of one or more second images with a first FOV of one or more first images, camera processor 14 is configured to cause a second image sensor to crop the second FOV to match the first FOV. In another example, in order to further synchronize the second FOV of one or more second images with the first FOV of one or more first images, camera processor 14 is also configured to cause the second image sensor to spatially align (e.g., recenter) the second FOV to match the first FOV.

[0117] In another example of this disclosure, in order to synchronize the second resolution of one or more second images with the first resolution of one or more first images, the camera processor 14 is further configured to cause the second image sensor to perform re-mosaicing on one or more second images having a synchronized second FOV. In another example, in order to synchronize the second resolution of one or more second images with the first resolution of one or more first images, the camera processor 14 is further configured to cause the second image sensor to perform a merge readout operation on one or more second images having a synchronized second FOV.

[0118] In another example of this disclosure, in order to generate a first HDR image from one or more first images, the camera processor 14 is also configured to use a multi-frame HDR technique to generate the first HDR image using multiple first images from the one or more first images. In one example, in order to perform tone alignment on one or more second images based on the first HDR image to generate a second HDR image, the camera processor 14 is also configured to use a multi-frame HDR technique to generate the second HDR image using multiple second images from the one or more second images.

[0119] In another example, to generate a first HDR image from one or more first images, camera processor 14 is also configured to use a first HDR technique to generate the first HDR image using multiple first images from the one or more first images. In this example, to perform tone alignment on one or more second images based on the first HDR image to generate a second HDR image, camera processor 14 is also configured to use a second HDR technique to generate the second HDR image, which approximates the tonal range generated by the first HDR technique. In one example, the first HDR technique is a multi-frame HDR technique, and the second HDR technique is either a triple-exposure HDR technique or an adaptive dynamic range coding (ADRC) technique.

[0120] Other illustrative examples of this disclosure are listed below.

[0121] Example 1 - An apparatus configured for camera processing, the apparatus comprising: means for receiving one or more first images from a first image sensor, wherein the one or more first images have a first field of view (FOV) and a first resolution; means for generating a first high dynamic range (HDR) image from the one or more first images; means for generating a second high dynamic range (HDR) image; and means for generating a depth map using the first HDR image and the second HDR image.

[0122] Example 2 - The apparatus as described in Example 1 further includes: components for receiving one or more second images from a second image sensor, wherein the one or more second images have a second FOV and a second resolution; and components for generating the second HDR image from the one or more second images.

[0123] Example 3 - An apparatus as described in Example 2, wherein the component for generating the second HDR image from the one or more second images includes: a component for performing tone alignment on the one or more second images based on the first HDR image to generate the second HDR image.

[0124] Example 4 - The apparatus as described in Example 2 further includes: a component for synchronizing a second FOV of one or more second images with a first FOV of one or more first images; and a component for synchronizing a second resolution of one or more second images with a first resolution of one or more first images, wherein the component for generating a second HDR image from one or more second images appears after the component for synchronizing the second FOV and the component for synchronizing the second resolution.

[0125] Example 5 - An apparatus as described in Example 4, wherein the components for synchronizing a second FOV of one or more second images with a first FOV of one or more first images include: components for causing a second image sensor to crop the second FOV to match the first FOV.

[0126] Example 6 - The apparatus as described in Example 5, wherein the component for synchronizing a second FOV of one or more second images with a first FOV of one or more first images further includes: a component for spatially aligning the second FOV to match the first FOV.

[0127] Example 7 - An apparatus as described in Example 2, wherein the component for synchronizing a second resolution of one or more second images with a first resolution of one or more first images includes: a component for causing a second image sensor to perform re-mosaic on one or more second images having a synchronized second FOV.

[0128] Example 8 - An apparatus as described in Example 2, wherein the component for synchronizing a second resolution of one or more second images with a first resolution of one or more first images includes: a component for causing a second image sensor to perform a merge readout operation on one or more second images having a synchronized second FOV.

[0129] Example 9 - An apparatus as described in Example 2, wherein the component for generating a first HDR image from one or more first images includes a component for generating the first HDR image using a plurality of first images from one or more first images using a multi-frame HDR technique, and wherein the component for generating a second HDR image from one or more second images includes a component for generating the first HDR image using a plurality of second images from one or more second images using the multi-frame HDR technique.

[0130] Example 10 - An apparatus as described in Example 2, wherein the component for generating a first HDR image from one or more first images includes a component for generating the first HDR image using a plurality of first images from one or more first images with a first HDR technique, and wherein the component for generating a second HDR image from one or more second images includes a component for generating the first HDR image using a second HDR technique based on a tonal range produced by the first HDR technique.

[0131] Example 11 - An apparatus as described in Example 2, wherein the first HDR technique is a multi-frame HDR technique, and wherein the second HDR technique is one of a triple-exposure HDR technique or an adaptive dynamic range coding (ADRC) technique.

[0132] Example 12 - The apparatus as described in Example 1 further includes: a component for applying an image effect to a first HDR image using a depth map.

[0133] Example 13 - A device as described in Example 12, wherein the image effect is a defocus effect.

[0134] In one or more examples, the described functionality can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality can be stored on or transmitted via a computer-readable medium as one or more instructions or code, and executed by a hardware-based processing unit. A computer-readable medium can include a computer-readable storage medium, which corresponds to a tangible medium such as a data storage medium. In this way, a computer-readable medium can substantially correspond to a non-transitory tangible computer-readable storage medium. A data storage medium can be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products can include computer-readable media.

[0135] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, cache memory, or any other media that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. It should be understood that computer-readable storage media and data storage media do not include carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the foregoing should also be included within the scope of computer-readable media.

[0136] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0137] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to highlight functional aspects of a device configured to perform the disclosed techniques, but they do not necessarily have to be implemented by different hardware units.

[0138] Various examples have been described. These and other examples are all within the scope of the appended claims.

Claims

1. A device configured for camera processing, the device comprising: A memory configured to receive images; as well as One or more processors that communicate with the memory, the one or more processors being configured to: Receive one or more first images from a first image sensor, wherein the one or more first images have a first field of view (FOV) and a first resolution; Generate a first high dynamic range (HDR) image from the one or more first images; Receive one or more second images from a second image sensor, wherein the one or more second images have a second field of view and a second resolution; Generating a second high dynamic range (HDR) image from the one or more second images, including performing tone alignment on the one or more second images based on the first HDR image; and A depth map is generated using the first HDR image and the second HDR image.

2. The device according to claim 1, wherein, The one or more processors are further configured to: Synchronize the second FOV of the one or more second images with the first FOV of the one or more first images; as well as Synchronize the second resolution of the one or more second images with the first resolution of the one or more first images, and Wherein, after synchronizing the second FOV and synchronizing the second resolution, the one or more processors generate the second HDR image from the one or more second images.

3. The device according to claim 2, wherein, In order to synchronize the second FOV of the one or more second images with the first FOV of the one or more first images, the one or more processors are further configured to: This causes the second image sensor to crop the second FOV to match the first FOV.

4. The device according to claim 3, wherein, To further synchronize the second FOV of the one or more second images with the first FOV of the one or more first images, the one or more processors are further configured to: Align the second FOV in space to match the first FOV.

5. The device according to claim 2, wherein, In order to synchronize the second resolution of the one or more second images with the first resolution of the one or more first images, the one or more processors are further configured to: This causes the second image sensor to perform re-mosaic on one or more second images having a synchronized second FOV.

6. The device according to claim 2, wherein, In order to synchronize the second resolution of the one or more second images with the first resolution of the one or more first images, the one or more processors are further configured to: This causes the second image sensor to perform a merge readout operation on one or more second images having a synchronized second FOV.

7. The device according to claim 1, wherein, In order to generate the first HDR image from the one or more first images, the one or more processors are further configured to use a multi-frame HDR technique to generate the first HDR image using multiple first images from the one or more first images, and In order to generate the second HDR image from the one or more second images, the one or more processors are further configured to use the multi-frame HDR technology to generate the second HDR image using multiple second images from the one or more second images.

8. The device according to claim 1, wherein, In order to generate the first HDR image from the one or more first images, the one or more processors are further configured to use a first HDR technique to generate the first HDR image using multiple first images from the one or more first images, and In order to generate the second HDR image from the one or more second images, the one or more processors are further configured to use a second HDR technique to generate the second HDR image based on the tonal range generated by the first HDR technique.

9. The device according to claim 8, wherein, The first HDR technology is a multi-frame HDR technology, and the second HDR technology is either a triple-exposure HDR technology or an adaptive dynamic range coding (ADRC) technology.

10. The device according to claim 1, wherein, The one or more processors are further configured to: The depth map is used to apply image effects to the first HDR image.

11. The device according to claim 10, wherein, The image effect described is a defocusing effect.

12. The device according to claim 1, further comprising: The first image sensor; The second image sensor; as well as A display configured to display the first HDR image.

13. A method for camera processing, the method comprising: Receive one or more first images from a first image sensor, wherein the one or more first images have a first field of view (FOV) and a first resolution; Generate a first high dynamic range (HDR) image from the one or more first images; Receive one or more second images from a second image sensor, wherein the one or more second images have a second field of view and a second resolution; Generating a second high dynamic range (HDR) image from the one or more second images, including performing tone alignment on the one or more second images based on the first HDR image; and A depth map is generated using the first HDR image and the second HDR image.

14. The method of claim 13, further comprising: Synchronize the second FOV of the one or more second images with the first FOV of the one or more first images; as well as Synchronize all the second resolutions of the one or more second images with the first resolutions of the one or more first images, and Specifically, after synchronizing the second FOV and synchronizing the first resolution, the second HDR image is generated from the one or more second images.

15. The method according to claim 14, wherein, Synchronizing the second FOV of the one or more second images with the first FOV of the one or more first images includes: This causes the second image sensor to crop the second FOV to match the first FOV.

16. The method according to claim 15, wherein, Synchronizing the second FOV of the one or more second images with the first FOV of the one or more first images further includes: Align the second FOV in space to match the first FOV.

17. The method of claim 14, wherein, Synchronizing the second resolution of the one or more second images with the first resolution of the one or more first images includes: This causes the second image sensor to perform re-mosaic on one or more second images having a synchronized second FOV.

18. The method according to claim 14, wherein, Synchronizing the second resolution of the one or more second images with the first resolution of the one or more first images includes: This causes the second image sensor to perform a merge readout operation on one or more second images having a synchronized second FOV.

19. The method according to claim 13, wherein, Generating the first HDR image from the one or more first images includes using multi-frame HDR technology to generate the first HDR image using multiple first images from the one or more first images, and Generating the second HDR image from the one or more second images includes using the multi-frame HDR technique to generate the second HDR image using multiple second images from the one or more second images.

20. The method according to claim 13, wherein, Generating the first HDR image from the one or more first images includes generating the first HDR image using a first HDR technique with multiple first images from the one or more first images, and Generating the second HDR image from the one or more second images includes generating the second HDR image using a second HDR technique based on the tonal range generated by the first HDR technique.

21. The method according to claim 20, wherein, The first HDR technology is a multi-frame HDR technology, and the second HDR technology is either a triple-exposure HDR technology or an adaptive dynamic range coding (ADRC) technology.

22. The method of claim 13, further comprising: The depth map is used to apply image effects to the first HDR image.

23. The method according to claim 22, wherein, The image effect described is a defocusing effect.

24. An apparatus configured to perform camera processing, the apparatus comprising: A component for receiving one or more first images from a first image sensor, wherein the one or more first images have a first field of view (FOV) and a first resolution; Components for generating a first high dynamic range (HDR) image from the one or more first images; A component for receiving one or more second images from a second image sensor, wherein the one or more second images have a second field of view (FOV) and a second resolution; For generating a second high dynamic range (HDR) image from the one or more second images, including components for performing tone alignment on the one or more second images based on the first HDR image; and A component for generating a depth map using the first HDR image and the second HDR image.

25. A non-transitory computer-readable storage medium storing instructions, which, when executed, cause one or more processors to: Receive one or more first images from a first image sensor, wherein... The one or more first images have a first field of view (FOV) and a first resolution; Generate a first high dynamic range (HDR) image from the one or more first images; Receive one or more second images from a second image sensor, wherein the one or more second images have a second field of view and a second resolution; Generating a second high dynamic range (HDR) image from the one or more second images includes performing tone alignment on the one or more second images based on the first HDR image; as well as A depth map is generated using the first HDR image and the second HDR image.

26. A computer program product comprising computer-readable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 13 to 23.

Images