Synchronizing image signal processing across multiple image sensors
By using image statistical information comparison and matching in head-mounted electronic devices to synchronize the processing of multiple image sensors, the problem of image mismatch captured by multiple image sensors is solved, and the user experience and image consistency are improved.
Patent Information
- Application Number
- CN202410609049.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-02-21
- Filing Date
- 2024-05-16
- Publication Date
- 2024-11-22
AI Technical Summary
Designing a head-mounted electronic device with multiple cameras When accidentally used, the captured images may have mismatched settings, resulting in visual discomfort, and prior art is difficult to synchronize images captured by multiple image sensors to avoid conflicts.
By capturing images with image sensors with different fields of view in electronic devices, and comparing and matching using image statistics, processing circuits are used to synchronize the processing of image sensors to ensure that the image settings displayed to users are consistent. Specific methods include comparing thumbnail information, brightness information, color information and focus information, and automatically adjusting exposure, white balance and tone mapping to match at a per-pixel level or global level.
Effectively synchronize images captured by multiple image sensors, avoid mismatched image settings, improve user experience, and ensure the consistent and quality of displayed images.
Smart Images

Figure CN119011747A_ABST
Abstract
Description
[0001] This application claims priority to U.S. Patent Application No. 18 / 583,729, filed on February 21, 2024, which claims the benefit of U.S. Provisional Patent Application No. 63 / 503,136, filed on May 18, 2023, which are hereby incorporated by reference in their entirety. Technical Field
[0002] The present disclosure relates generally to electronic devices, and more particularly to electronic devices such as head mounted devices. Background Art
[0003] An electronic device such as a head mounted device may have a camera for capturing a video feed of an external scene, and one or more displays for presenting the video feed of the captured scene to a user. The head mounted device may include a hardware or software subsystem for processing the video feed, such as a hardware / software subsystem for applying image quality adjustments on the captured video feed.
[0004] Designing a head mounted device with multiple cameras can be challenging. If you are not careful, the images captured using multiple cameras may have mismatched settings. Displaying images with mismatched settings to the user may cause visual discomfort. It is in this context that the embodiments of this article are generated. Summary of the invention
[0005] An electronic device, such as a head mounted device, may include one or more cameras for capturing a video feed of a real world environment and one or more displays for presenting a pass-through video feed to a user. The electronic device may include processing circuitry for performing one or more processing functions on the captured video feed to generate the pass-through video feed.
[0006] One aspect of the present disclosure provides a method of operating an electronic device having at least a first image sensor and a second image sensor, the method comprising: using the first image sensor having a first field of view to capture a first image; using the second image sensor having a second field of view different from the first field of view to capture a second image; determining whether the first image sensor is currently obscured; and in response to determining that the first image sensor is currently obscured, synchronizing the processing of the first image and the second image by processing the first image and the second image using information associated with the second image. The method may also include: displaying the first processed image using a first display in the electronic device and displaying the second processed image using a second display in the electronic device.
[0007] The operation for determining whether the first image sensor is currently obscured may include: comparing thumbnail information associated with the first image with thumbnail information associated with the second image; comparing brightness information associated with the first image with brightness information associated with the second image; comparing color information associated with the first image with color information associated with the second image; comparing focus information associated with the first image with focus information associated with the second image; and / or comparing a first integration time associated with the first image with a second integration time associated with the second image. The operation for synchronizing the processing of the first image and the second image may include: synchronizing automatic exposure of the first image and the second image using brightness information associated with the second image; synchronizing automatic white balance of the first image and the second image using color information associated with the second image; synchronizing tone mapping of the first image and the second image using histogram information associated with the second image; and / or synchronizing one or more additional image signal processing functions of the first image and the second image using pixel information associated with the second image.
[0008] One aspect of the present disclosure provides a method of operating an electronic device having at least a first image sensor and a second image sensor, the method comprising: capturing a first image using a first image sensor having a first field of view; capturing a second image using the second image sensor having a second field of view different from the first field of view, wherein the first field of view and the second field of view coincide in an overlapping field of view (FOV) region; and processing the first image and the second image using one or more combined image signal processing control parameters calculated based on information associated with the first image in the overlapping FOV region and information associated with the second image in the overlapping FOV region. The method may include: facilitating color, tone mapping, brightness and / or noise matching between the first image and the second image. The method may include: performing spatial matching at a per-pixel level and / or performing global matching for the entire image. The method may also include: processing the first image and the second image using additional combined image signal processing control parameters calculated based on information associated with the first image and the second image in a total field of view (FOV) region equal to the union of the first field of view and the second field of view.
[0009] One aspect of the present disclosure provides an apparatus comprising: a first image sensor having a first field of view and configured to capture a first video feed; a second image sensor having a second field of view different from the first field of view and configured to capture a second video feed; an image statistics collection subsystem configured to collect first image statistics associated with the first video feed and to collect second image statistics associated with the second video feed; an image capture disparity detection subsystem configured to compare the first image statistics with the second image statistics; and a plurality of image signal processing blocks that synchronize processing of the first video feed and the second video feed based on the comparison of the first image statistics with the second image statistics. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 is a top view of an illustrative head mounted device according to some embodiments.
[0011] Figure 2 is a schematic diagram of an illustrative electronic device according to some embodiments.
[0012] Figure 3 is a diagram showing that according to some embodiments Figure 1 or Figure 2 An illustration of how the electronic device shown in can be used to capture at least two images with partially overlapping fields of view.
[0013] Figure 4 is a diagram of an illustrative electronic device with multiple image sensors coupled to an image statistics collection block and an image signal processor according to some embodiments.
[0014] Figure 5 According to some embodiments, Figure 4 Flowchart of illustrative steps for an electronic device of the type shown in .
[0015] Figure 6 is a diagram of an illustrative spatial weight map according to some embodiments. DETAILED DESCRIPTION
[0016] Figure 1 A top view of an exemplary head mounted device is shown in FIG. Figure 1As shown in , a head-mounted device such as electronic device 10 may have a head-mounted support structure such as housing 12. Housing 12 may include a portion (e.g., head-mounted support structure 12T) for allowing device 10 to be worn on the user's head. Support structure 12T may be formed of fabric, polymer, metal, and / or other materials. Support structure 12T may form a strap or other head-mounted support structure to help support device 10 on the user's head. The main support structure of housing 12 (e.g., a head-mounted housing such as main housing portion 12M) may support electronic components such as display 14.
[0017] The main housing portion 12M may include a housing structure formed of metal, polymer, glass, ceramic, and / or other materials. For example, the housing portion 12M may have a housing wall on the front side F and housing walls on adjacent top, bottom, left, and right sides formed of a rigid polymer or other rigid support structure, and these rigid walls may be optionally covered with electronic components, fabric, leather, or other soft materials, etc. The housing portion 12M may also have an internal support structure, such as a frame (chassis) and / or a structure that performs multiple functions (such as controlling airflow and heat dissipation while providing structural support).
[0018] The walls of housing portion 1254 may enclose internal components 38 in interior region 34 of device 10, and may separate interior region 34 from the environment (exterior region 36) surrounding device 10. Internal components 38 may include integrated circuits, actuators, batteries, sensors, and / or other circuits and structures for device 10. Housing 12 may be configured to be worn on a user's head, and may form glasses, spectacles, a hat, a mask, a helmet, goggles, and / or other head-mounted devices. Configurations in which housing 12 forms goggles are sometimes described herein as an example.
[0019] The front side F of the housing 12 may face outwardly, away from the user's head and face. The opposite rear side R of the housing 12 may face the user. The portion of the housing 12 located on the rear side R (e.g., the portion of the main housing 12M) may form a cover, such as a cover 12C (sometimes referred to as a blind). The presence of the cover 12C on the rear side R may help to hide the internal housing structure, the interior components 38, and other structures in the interior area 34 from being seen by the user.
[0020] Device 10 may have one or more cameras, such as Figure 1Camera 46. Camera 46 mounted on front face F and facing outward (toward the front of device 10 and away from the user) may sometimes be referred to herein as a forward-pointing or front-facing camera. Camera 46 may capture visual odometry information, image information processed to locate objects in the user's field of view (e.g., so that virtual content can be properly registered relative to real-world objects), image content displayed in real time to the user of device 10, and / or other suitable image data. For example, a forward-pointing (positive) camera may allow device 10 to monitor movement of device 10 relative to the environment surrounding device 10 (e.g., the camera may be used to form part of a visual odometry system or a visual-inertial odometry system). The forward-pointing camera may also be used to capture images of the environment displayed to the user of device 10. If desired, images from multiple forward-pointing cameras may be merged with each other and / or the forward-pointing camera content may be merged with computer-generated content for the user.
[0021] Device 10 may have any suitable number of cameras 46. For example, device 10 may have K cameras, where the value of K is at least one, at least two, at least four, at least six, at least eight, at least ten, at least 12, less than 20, less than 14, less than 12, less than 10, 4-10, or other suitable value. Cameras 46 may be sensitive at infrared wavelengths (e.g., camera 46 may be an infrared camera), may be sensitive at visible wavelengths (e.g., camera 46 may be a visible camera), and / or cameras 46 may be sensitive at other wavelengths. If desired, cameras 46 may be sensitive at both visible and infrared wavelengths.
[0022] The device 10 may have a left optical module and a right optical module 40. The optical module 40 supports electronic components and optical components such as light-emitting components and lenses, and therefore may sometimes be referred to as an optical assembly, an optical system, an optical component support structure, a lens and display support structure, an electronic component support structure, or a housing structure. Each optical module may include a corresponding display 14, a lens 30, and a support structure such as a support structure 32. The support structure 32, which may sometimes be referred to as a lens support structure, an optical component support structure, an optical module support structure, or an optical module portion, or a lens barrel, may include a hollow cylindrical structure with an open end or other support structures for accommodating the display 14 and the lens 30. The support structure 32 may, for example, include a left lens barrel that supports the left display 14 and the left lens 30 and a right lens barrel that supports the right display 14 and the right lens 30.
[0023] Display 14 may include an array of pixels or other display devices to produce images. Display 14 may include, for example, organic light emitting diode pixels formed on a substrate having thin film circuits and / or formed on a semiconductor substrate, pixels formed from a crystalline semiconductor die, liquid crystal display pixels, scanning display devices, and / or other display devices for producing images.
[0024] Lens 30 may include one or more lens elements for providing image light from display 14 to corresponding eye zone 13. The lens may be implemented using a refractive glass lens element, using a mirror lens structure (a catadioptric lens), using a Fresnel lens, using a holographic lens, and / or other lens systems.
[0025] When the user's eyes are located in the eye zone 13, the display (display panel) 14 operates together to form the display of the device 10 (e.g., the user's eyes can view the images provided by the corresponding left and right optical modules 40 in the eye zone 13, so that a stereoscopic image is created for the user). When the user views the display, the left image from the left optical module is fused with the right image from the right optical module.
[0026] It may be desirable to monitor the user's eyes when the user's eyes are located in the eye-adaptive zone 13. For example, it may be desirable to use a camera to capture an image of the user's iris (or other parts of the user's eyes) for user authentication. It may also be desirable to monitor the direction of the user's gaze. Gaze tracking information can be used as a form of user input and / or can be used to determine where the image content resolution should be locally enhanced within the image in the fovea imaging system. In order to ensure that the device 10 can capture a satisfactory eye image when the user's eyes are located in the eye-adaptive zone 13, each optical module 40 may be provided with a camera (such as camera 42) and one or more light sources (such as light emitting diodes 44) or other light emitting devices (such as lasers, lamps, etc.). The camera 42 and the light emitting diode 44 can operate at any suitable wavelength (visible light, infrared light, and / or ultraviolet light). For example, the diode 44 can emit infrared light that is invisible (or almost invisible) to the user. This allows the eye monitoring operation to be performed continuously without interfering with the user's ability to view the image on the display 14.
[0027] Figure 2 A schematic diagram of an illustrative electronic device (such as a head-mounted device or other wearable device) is shown in FIG. Figure 2 Device 10 may operate as a standalone device and / or resources of device 10 may be used to communicate with external electronic equipment. For example, communication circuitry in device 10 may be used to transmit user input information, sensor information, and / or other information to an external electronic device (e.g., wirelessly or via a wired connection). Each of these external devices may include Figure 2 Components of the type shown in device 10.
[0028] like Figure 2As shown in , a head mounted device (such as device 10) may include a control circuit 20. The control circuit 20 may include storage and processing circuits for supporting the operation of the device 10. The storage and processing circuits may include storage devices, such as non-volatile memory (e.g., flash memory or other electrically programmable read-only memory configured to form a solid state drive), volatile memory (e.g., static or dynamic random access memory), etc. One or more processors in the control circuit 20 may be used to collect inputs from sensors and other input devices, and may be used to control output devices. The processing circuit may be based on one or more processors (such as microprocessors, microcontrollers, digital signal processors, baseband processors and other wireless communication circuits, power management units, audio chips, application specific integrated circuits, etc.). During operation, the control circuit 20 may use the display 14 and other output devices to provide visual output and other outputs to the user. The control circuit 20 may be configured to perform operations in the device 10 using hardware (e.g., dedicated hardware or circuits), firmware, and / or software. Software code for performing operations in the device 10 may be stored on a storage circuit (e.g., a non-transitory (tangible) computer-readable storage medium storing software code). The software codes may sometimes be referred to as program instructions, software, data, instructions or code. The stored software codes may be executed by processing circuitry within circuitry 20.
[0029] To support communication between device 10 and external equipment, control circuit 20 may communicate using communication circuit 22. Circuit 22 may include an antenna, radio frequency transceiver circuits, and other wireless communication circuits and / or wired communication circuits. Circuit 22 (which may sometimes be referred to as control circuits and / or control and communication circuits) may support two-way wireless communication between device 10 and external equipment (e.g., a companion device such as a computer, cellular phone, or other electronic device, an accessory such as a pointing device or controller, a computer stylus, or other input device, a speaker or other output device, etc.) via a wireless link.
[0030] For example, circuitry 22 may include radio frequency transceiver circuitry, such as wireless local area network transceiver circuitry configured to support communication via a wireless local area network link, near field communication transceiver circuitry configured to support communication via a near field communication link, cellular telephone transceiver circuitry configured to support communication via a cellular telephone link, or transceiver circuitry configured to support communication via any other suitable wired or wireless communication link. link, Wireless communications may be supported by a wireless link, a wireless link operating at a frequency between 10 GHz and 400 GHz, a 60 GHz link, or other millimeter wave link, a cellular telephone link, or other wireless communications link. Device 10 may include power circuitry for transmitting and / or receiving wired and / or wireless power, if desired, and may include a battery or other energy storage device. For example, device 10 may include a coil and a rectifier to receive wireless power provided to circuits in device 10.
[0031] Device 10 may include input-output devices such as device 24. Input-output devices 24 may be used to collect user input, to collect information about the environment surrounding the user, and / or to provide output to the user. Device 24 may include one or more displays such as display 14. Display 14 may include one or more display devices such as an organic light emitting diode display panel (a panel having organic light emitting diode pixels formed on a polymer substrate or silicon substrate containing pixel control circuitry), a liquid crystal display panel, a micro-electromechanical system display (e.g., a two-dimensional mirror array or scanning mirror display device), a display panel having an array of pixels formed from crystalline semiconductor light emitting diode dies (sometimes referred to as micro-LEDs), and / or other display devices.
[0032] Sensors 16 in input-output devices 24 may include force sensors (e.g., strain gauges, capacitive force sensors, resistive force sensors, etc.), audio sensors (such as microphones), touch sensors and / or proximity sensors (such as capacitive sensors, such as touch sensors forming buttons, trackpads, or other input devices), and other sensors. If desired, sensors 16 may include optical sensors (such as optical sensors that emit and detect light), ultrasonic sensors, optical touch sensors, optical proximity sensors, and / or other touch sensors and / or proximity sensors, monochrome ambient light sensors and color ambient light sensors, image sensors (e.g., cameras), fingerprint sensors, iris scan sensors, retina scan sensors, and other biometric sensors, temperature sensors, sensors for measuring three-dimensional contactless gestures (“air gestures”), pressure sensors, sensors for detecting the position, orientation, and / or movement of device 10 and / or information about the user's head posture (e.g., accelerometers, magnetic sensors such as Lo-Fi sensors, etc.), and / or proximity sensors. 10 may include a sensor 16 and / or other input-output devices to collect user input. For example, a button may be used to collect button press input, a touch sensor overlapped with a display may be used to collect user touch screen input, a touch pad may be used to collect touch input, a microphone may be used to collect audio input (e.g., voice commands), an accelerometer may be used to monitor when a finger contacts an input surface and may therefore be used to collect finger press input, etc.
[0033] To help protect the privacy of users, best practices may be used to handle any personal user information collected by sensors. These best practices include meeting or exceeding any applicable privacy regulations. Opt-in and opt-out options and / or other options may be provided that allow users to control the use of their personal data.
[0034] If desired, the electronic device 10 may include additional components (see, for example, other devices 18 in input-output devices 24). Additional components may include tactile output devices, actuators for moving the movable housing structure, audio output devices such as speakers, light emitting diodes for status indicators, light sources such as light emitting diodes that illuminate portions of the housing and / or display structure, other optical output devices, and / or other circuits for collecting input and / or providing output. The device 10 may also include a battery or other energy storage device, a connector port for supporting wired communication with auxiliary equipment and for receiving wired power, and other circuits.
[0035] Display 14 may be used to present various content to the user's eyes. When viewed through eye zone 13, left and right displays 14 used to present a fused stereoscopic image to the user's eyes may sometimes be collectively referred to as display 14. For example, display 14 may present virtual reality (VR) content. Virtual reality content may refer to content that includes only virtual objects in a virtual reality (computer-generated) environment. As another example, display 14 may present mixed reality (MR) content. Mixed reality content may refer to content that includes virtual objects and real objects from the real-world physical environment in which device 10 is being operated. As another example, display 14 may only present real-world content. Real-world content may refer to content captured by one or more front-facing cameras (see, e.g., Figure 1 The real-world content captured by the front-facing camera is thus sometimes referred to as a camera pass-through feed, a (live) video pass-through feed, or a pass-through video feed (stream).
[0036] A physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In the case of an XR system, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one law of physics.
[0037] Although Figure 1 and Figure 2 The electronic device 10 is sometimes referred to herein as a head-mounted electronic device, but the embodiments disclosed herein are not necessarily limited to head-mounted devices or wearable devices. In some embodiments, Figure 2 The device 10 may form part of a larger system having multiple outward-facing cameras. As an example, the device 10 may form part of a self-driving vehicle having multiple cameras and / or sensors.
[0038] Figure 3 It is shown Figure 1 or Figure 2 , which is a diagram of how the electronic device shown in is used to capture at least two images with partially overlapping fields of view (FOV). Figure 3 As shown in , device 10 may include at least a first front camera 46-1 and a second front camera 46-2. Cameras 46-1 and 46-2 may be color image sensors (e.g., sensors for acquiring color images of a scene) or monochrome image sensors (e.g., sensors for acquiring black and white images or grayscale images or scenes). First image sensor 46-1 may sometimes be referred to as a left (L) scene camera, and second image sensor 46-2 may sometimes be referred to as a right (R) scene camera.
[0039] The first image sensor 46-1 may be configured to capture a first image having a first field of view (FOV) as depicted by block 60, while the second image sensor 46-1 may be configured to capture a second image having a second FOV as depicted by block 62. The first FOV 60 may be slightly different from the second FOV 62. The first FOV 60 is sometimes referred to as the left FOV, while the second FOV 62 is sometimes referred to as the right FOV. The fields of view (FOV) of the first image and the second image captured by the two image sensors 46-1 and 46-2 may at least partially overlap. Figure 3 As shown in , a first FOV 60 associated with a first image captured by a first image sensor 46-1 and a second FOV 62 associated with a second image captured by a second image sensor 46-2 may overlap, intersect, or share a common area 64. Area 64 is sometimes referred to and defined herein as an overlapping field of view (FOV) or overlapping FOV area. There may be portions of the left FOV 60 that do not overlap with the right FOV 62. For example, at least the leftmost portion (area) 66 of the left FOV 60 does not overlap with the right FOV 62, and therefore may contain information that is not captured by the right image sensor 46-2. Similarly, there may be portions of the right FOV 62 that do not overlap with the left FOV 60. For example, at least the rightmost portion (area) 68 of the right FOV 62 does not overlap with the left FOV 60, and therefore may contain information that is not captured by the left image sensor 46-1. The union of the first FOV 60 and the second FOV 62 (which is the same as the overlapping FOV area 64 and the union of the non-overlapping FOV area 66 and the non-overlapping FOV area 68) may be referred to and defined herein as a total FOV area.
[0040] Designing an electronic device 10 with multiple image sensors can be challenging. If care is not taken, images captured from the multiple image sensors may have mismatched settings. Displaying images with mismatched settings to a user may cause visual discomfort. Therefore, it would be desirable to synchronize images captured from the multiple image sensors so that images with conflicting image settings are not presented to the user's eyes.
[0041] Figure 4 is a diagram of an illustrative electronic device 10 having multiple cameras, such as image sensor 46-1 and image sensor 46-2, coupled to one or more subsystems configured to synchronize image signal processing control parameters to generate a blended video or image from the multiple image sensors. Figure 4 As shown in , image sensor 46-1 may be configured to acquire or capture a first image of a given scene (sometimes collectively referred to as a first video feed), while image sensor 46-2 may be configured to acquire or capture a second image of a given scene (sometimes collectively referred to as a second video feed).
[0042] Where the device 10 includes two image sensors 46-1 and 46-2 for capturing multiple video feeds to be displayed Figure 4 The examples are illustrative. In general, device 10 may include two or more image sensors 46 for capturing at least two video feeds to be synchronized, two to five image sensors 46 for capturing up to five video feeds to be synchronized, five to ten image sensors 46 for capturing up to ten video feeds to be synchronized, 10 to 20 image sensors 46 for capturing up to 20 video feeds to be synchronized, or more than 20 image sensors 46 for capturing more than 20 video feeds to be synchronized. Illustrative configurations in which images from two image sensors 46-1 and 46-2 are being synchronized are sometimes described herein as examples.
[0043] The image sensor 46-1 may output a first image (see image 1) to an image statistics collection subsystem such as an image statistics collection block 70 and also output it to an image signal processing circuit such as an image signal processor (ISP) 80. Similarly, the image sensor 46-2 may output a second image (see image 2) to the image statistics collection block 70 and also output it to the image signal processor 80. The first image and the second image directly output from the image sensor are sometimes referred to as unprocessed images, straight out of camera (SOOC) images, or raw images. The first image and the second image may be analyzed by the image statistics collection block 70, which is sometimes referred to as an image statistics analyzer. The image statistics collection block 70 may receive a captured image from the image sensor 46 and analyze the captured image to obtain information related to the captured image, such as thumbnail information, integration time, brightness histogram, color statistics, focus statistics, and / or other statistical information related to the captured image. The image statistics collection block 70 may independently collect image frame statistics from the first image and the second image.
[0044] The thumbnail information may include low-resolution versions of the first image and the second image. The integration time may include a first integration time for acquiring the first image, which may be a function of the shutter speed of the image sensor 46-1, and may also include a second integration time for acquiring the second image, which may be a function of the shutter speed of the image sensor 46-2. The brightness histogram may include local and / or global histograms of the first image and the second image. The brightness histogram plots the frequency of image pixels across a range of possible brightness values. The color statistics of each captured image may include a color histogram (e.g., a graph plotting the distribution of chromaticity values in an image), an average color value (e.g., an average of all camera red values, green values, and blue values in an image), and / or other color-related information. The focus statistics may include a first focus setting for capturing the first image and may include a second focus setting for capturing the second image. The first focus setting and the second focus setting may be auto focus (AF) settings or manual focus (MF) settings. The focus statistics may also indicate whether the image sensor is able to lock focus. The information collected by the image statistics collection block 70 may be collectively referred to as image statistics information.
[0045] The image statistics information obtained at the image statistics collection block 70 and the raw image output from the image sensor 26 may be fed to an image signal processor 80. The image signal processor 80 is sometimes considered to be the control circuit 20 ( Figure 2) is an image processing unit for processing raw digital images output from one or more cameras, such as image sensor 46-1 and image sensor 46-2, on device 10. Image signal processor 80 may be used to perform a series of post-processing operations on the captured images before displaying the images to a user of device 10. Image signal processor 80 may process the raw images to produce corresponding processed images, which may then be presented on one or more displays 14. Image statistics collection block 70, image signal processor 80, display 14, and / or other associated subsystems in the signal path are sometimes referred to as part of a display pipeline. The images captured by image sensor 46-1 and image sensor 46-2 are shown as being transmitted to a single ISP 80. Figure 4 The configuration is exemplary. In other embodiments, images captured from multiple image sensors may be fed to separate image signal processors 80 (e.g., a first image captured by sensor 46-1 may be processed using a first ISP 80, while a second image captured by sensor 46-2 may be processed using a second ISP 80).
[0046] like Figure 4 As shown in the example of , the image signal processor 80 may include an automatic exposure (AE) block 82 for controlling the exposure setting of the captured image, a color correction block 84 (sometimes referred to as an automatic white balance or AWB block) for controlling the white balance setting of the captured image, a tone mapping block 86 (e.g., a global tone mapping block for globally controlling the tone curve setting of the captured image and / or a local tone mapping block for locally controlling the tone curve setting of one or more portions of the captured image), an auto focus (AF) block 88 for controlling the focus setting for capturing the image, and other image signal processing blocks. Figure 4Other image signal processing blocks in the ISP 80 of the present embodiment may include a gamma correction block for controlling the gamma curve setting of the image, a shading correction block for controlling how the image sensor corrects for regional brightness variations in the captured image, a noise reduction block for controlling how the image sensor reduces noise in the captured image, a black level adjustment block for controlling the black level setting for the image sensor, a demosaicing block for controlling how the image sensor reconstructs a full color image from a Bayer color pattern (as an example), an image sharpening block, a high dynamic range (HDR) correction block, a facial recognition block for detecting faces in a scene, an automatic scene recognition block for identifying different types of scenery, a lens correction block for correcting lens shading problems, lens vignetting, and lens distortion, a color space conversion block for converting RGB signals to other color spaces (if necessary), and / or other image signal processing blocks. These individual blocks within the ISP 80 may be configured to run their own image processing algorithms and are sometimes collectively referred to as image signal processing blocks.
[0047] In some embodiments, a first image captured by image sensor 46-1 may be processed using ISP 80 and presented on first (left) display 14 of device 10, while a second image captured by image sensor 46-2 may be processed using ISP 80 and presented on second (right) display 14 (see, e.g., Figure 1 ). The image signal processing blocks (e.g., blocks 82, 84, 86, 88, etc.) of the ISP 80 should apply the same image adjustments to the first image and the second image to ensure that the processed images output on the left and right displays 14 appear harmonious to the user. The image adjustments or settings applied to the first image and the second image may be determined based on the image statistics collected by the image statistics collection block 70. However, the image statistics of the first image and the second image may not always be the same, and may differ significantly in some cases.
[0048] In order to detect such differences in image statistics between the first image and the second image, the ISP 80 may be provided with a disparity detection subsystem, such as an image capture disparity detection block 90. The image capture disparity detection block 90 may be configured to detect differences or errors between the first image and the second image or video feed, and may be further configured to use image statistics associated with a selected one of the first image and the second image to simultaneously adjust the two images. Image adjustments or settings (such as automatic exposure settings, color correction (AWB) settings, tone mapping settings, and / or other control parameters) applied to captured images during operation of the ISP 80 to ensure that the first image is properly matched to the second image are sometimes collectively referred to as "image signal processing control parameters." Thus, the image capture disparity detection block 90 may facilitate synchronizing image signal processing control parameters between left and right images captured by the image sensor 46-1 and the image sensor 46-2. Where the disparity detection block 90 is shown as part of the ISP 80 Figure 4 The example is for illustration only. In other embodiments, the image capture disparity detection block 90 may be considered as part of the image statistics collection block 70, which may be considered as Figure 2 It may be part of the control circuit 20 in, or it may be considered as an independent block separate from block 70 and ISP 80.
[0049] Figure 5 It is used to operate the combination Figure 4 1 and 10. During the operation of block 100, multiple cameras within device 10 may be configured to capture corresponding images or video frames. For example, first (left) image sensor 46-1 may be used to capture images with respect to the first field of view (see Figure 3 60) in the first image, while the second (right) image sensor 46-2 can be used to acquire a first image associated with the second field of view (see Figure 3 The fields of view of image sensor 46-1 and image sensor 46-2 at least partially overlap (see, for example, Figure 3 The overlapping FOV area 64).
[0050] During operation of block 102, the captured image may be fed to the image statistics collection block 70, and the image statistics collection block 70 may generate image statistics about the captured image. For example, the image statistics collection block 70 may generate a first thumbnail of the first image, identify a first integration time for the first image, obtain a local / global brightness histogram for the first image, obtain color statistics associated with the first image, and obtain focus statistics associated with the first image. Similarly, the image statistics collection block 70 may generate a second thumbnail of the second image, identify a second integration time for the second image, obtain a local / global brightness histogram for the second image, obtain color statistics associated with the second image, and obtain focus statistics associated with the second image. The fields of view of image sensor 46-1 and image sensor 46-2 at least partially overlap (e.g., see Figure 3 The overlapping FOV area 64 in the image thus may be similar, identical, or completely different image statistics of the first image and the second image.
[0051] During the operation of block 102, a spatial weighting map (such as Figure 6 The spatial weight map 200 of the first image and the second image is optionally processed. Figure 6 As shown in FIG. 1 , the spatial weight map 200 divides the image into a plurality of regions 202. Each region 202 in the spatial weight map 200 may be assigned a corresponding weight value. These weight values may be applied to the information in the corresponding region of the image to adjust the overall image statistics in a segmented manner. Each region 202 may have a square shape, a rectangular shape, or other shapes. The spatial weight map 200 divides the image into five rows and eight columns of regions 202. Figure 6 The examples are merely illustrative. In general, the spatial weight map 200 can be divided into any number of regions 202 of the same size / shape or different sizes / shapes.
[0052] As an example, a region 202 closer to the center of the spatial map or image may be assigned a greater weight, while a region 202 farther from the center may be assigned progressively lower weights (e.g., the lowest weights may be assigned to regions 202 along the periphery of the image). As another example, a region closer to a gaze point (such as a Figure 6 A larger weight may be assigned to an area 202 that is closer to the gaze point 204 in the example of FIG. 204, while a progressively lower weight may be assigned to an area 202 that is farther away from the gaze point 204. As another example, a weight may be assigned to an area that is closer to a bright light (such as, Figure 6206), while progressively lower weights may be assigned to areas 202 that are further away from the point 206, or the lowest weights may be assigned to areas 202 that correspond to darker areas of the image. These examples are illustrative. Other weighting heuristics may be employed if desired. The spatial weight maps 200 applied to the image statistics collected from the left and right images may be the same or may be different.
[0053] During the operation of block 104 (see Figure 5 ), the image capture disparity detection block 90 may be used to detect whether one of the image sensors is currently obscured or blocked based on image statistics of the first image and the second image. For example, consider the following situation: image sensor 46-1 is currently obscured by the user's hand (e.g., the FOV 60 of the left camera is primarily covered by the user's hand, arm, phone, or other obstruction) while image sensor 46-2 is not obscured (e.g., the FOV 62 of the right camera is not obscured by any nearby objects and can properly capture the scene in front of device 10). In this case, the first image captured by image sensor 46-1 may be substantially darker than the second image captured by image sensor 46-2. As an example, the disparity may be evident in the brightness histograms of the two images (e.g., the brightness histogram of the first image will be skewed toward darker values relative to the brightness histogram of the second image). As another example, the disparity may be evident in the thumbnails of the two images (e.g., the thumbnail of the first image will have a much smaller dynamic range and much less scene content relative to the thumbnail of the second image). As another example, the parallax may be evident in the integration times of the two images (e.g., the integration time used to capture the first, darker image will be substantially longer than the integration time used to capture the second image). As another example, the parallax may be evident in the color information of the two images (e.g., the color statistics of the first image may be softer relative to the color statistics of the second image). As another example, the parallax may be evident in the focus information associated with the two images (e.g., the first image sensor may not be able to achieve focus when obscured, while the second image sensor will be able to lock focus on a portion of the scene (assuming an autofocus function is enabled).
[0054] If desired, the image capture parallax detection block 90 may optionally compare past (historical) video frames with the current video frame to determine whether a new obstacle has recently entered the field of view of at least one of the image sensors. The above example in which block 90 is used to detect hand occlusion of one of the image sensors is illustrative only. In general, the above techniques may be used to detect whether one of the cameras is covered in mud, detect lens dirt on one of the cameras, and / or detect other types of occlusions or phenomena that may cause a captured image from one of the cameras to be temporarily erroneous. In other embodiments, a near-field object detection component may be additionally or alternatively employed to sense nearby objects, such as a user's hand near or in the FOV of one or more of the image sensors 46.
[0055] In response to determining that one of the image sensors is currently obscured, processing may proceed to block 106. During operation of block 106, the ISP 80 may adjust the first image and the second image using only the image statistics associated with the unobscured image sensor. As an example, the AE block 82 may apply an automatic exposure setting based only on thumbnail information or a luminance histogram of the image associated with the unobscured image sensor to both the first image and the second image (while completely ignoring the thumbnail information or luminance histogram of the image captured by the obscured image sensor). As another example, the AWB block 84 may apply a color correction based only on a color histogram of the image associated with the unobscured image sensor to both the first image and the second image (while completely ignoring the color histogram of the image captured by the obscured image sensor). As another example, the tone mapping block 82 may apply a tone curve setting based only on a luminance histogram of the image associated with the unobscured image sensor to both the first image and the second image (while completely ignoring the luminance histogram of the image captured by the obscured image sensor). These examples of synchronizing auto-exposure, auto-white balance, and tone mapping settings between the first image and the second image are illustrative. In general, one or more other image signal processing functions (such as demosaicing and denoising) within the ISP block 80 can also be synchronized using pixel information obtained from the unobstructed image sensor.
[0056] The above example in which image statistics from an obscured image sensor are completely ignored or discarded is illustrative. If desired, ISP 80 may optionally adjust the first image and the second image based on average image statistics of the overlapping FOV area. As an example, AE block 82 may adjust the first image and the second image based on the overlapping FOV area 64 of the two images (e.g., see Figure 3) to apply the automatic exposure setting based on the average of the thumbnail information or brightness histogram in the overlapping FOV area 64 of the two images, while ignoring the thumbnail or brightness information in the non-overlapping area 66 and the non-overlapping area 68. As another example, the AWB block 84 may apply color correction based on the average of the color histogram in the overlapping FOV area 64 of the two images, while ignoring the color information in the non-overlapping portion 66 and the non-overlapping portion 68. During the operation of block 108, the device 10 may optionally issue a notification to alert the user that one of the image sensors is obscured. The user may then be given the opportunity to remove or eliminate the obstruction (e.g., by moving his / her hand, by turning his / her head in different directions, by wiping mud or lens dirt from the obscured camera, etc.).
[0057] In some embodiments, in response to determining that one of the image sensors is currently obscured, a portion of the processed image output from the currently obscured image sensor can be filled with a corresponding portion of the processed image output from the image sensor that is not obscured (e.g., using information from another image sensor to fill the obscured image portion so that the processed image output from the currently obscured image sensor displays a non-black image). The examples described herein in which one of the images includes information lost due to occlusion are exemplary. In other embodiments, such lost information can be alternatively or additionally attributed to a failure of one or more components in the image processing pipeline.
[0058] Referring back to block 104, in response to determining that no image sensor is currently obscured, processing may proceed to block 110. During operation of block 110, combined or fused image statistics from the total FOV area may be considered for certain image signal processing control parameters (e.g., see Figure 3The image signal processing control parameters of interest from the total FOV may be averaged or weighted summed and applied to both the first image and the second image. Using a weighted sum of the image signal processing control parameters from the two images may help bias the resulting image toward information obtained from one of the image sensors. As an example, information from the left image may be given a 60% weight, while information from the right image may be given a 40% weight. In general, any weighting scheme may be used to mix information / statistics from the left and right images together. Using information from the total FOV covers the maximum amount of scene information, which may be beneficial for certain automatic white balance algorithms such as illuminant estimation (e.g., estimating the type of light source in the scene). The term "illuminant" may refer to the type of light source detected within a scene or captured image. Therefore, image signal processing functions such as automatic white balance may benefit from utilizing illumination information from the combined (total) FOV. The estimated illuminant type may, for example, be a single light source for global white balance correction of the first and second images. As another example, the estimated illuminant type may include multiple light sources for spatially varying white balance correction of the first image and the second image. At least some of the blocks within ISP 80 may run one or more algorithms that use one or more image signal processing control parameters that fall into this category.
[0059] During the operation of block 112, other image signal processing functions may take into account the image signal from overlapping FOVs (e.g., see Figure 364 in the image sensor) for improved color, tone mapping, brightness and / or noise matching (as an example). For example, an average or weighted sum of image signal processing control parameters of interest from overlapping FOVs may be calculated and applied to both the first image and the second image. Using a weighted sum of image signal processing control parameters from two images may help bias the resulting image toward information obtained from one of the image sensors. As an example, information from the left image may be given a 10% weight, while information from the right image may be given a 90% weight. In general, any weighting scheme may be used to blend information / statistics from the left and right images together. Using information only from the overlapping FOV region while ignoring the non-overlapping image portion 66 and the non-overlapping image portion 68 may be beneficial for certain algorithms such as color matching (e.g., matching the colors of the first image and the second image). Thus, image signal processing control parameters that are color matching parameters may benefit from utilizing only color information from the overlapping FOV region. At least some of the blocks within the ISP 80 may run one or more algorithms involving image signal processing control parameters that fall into this category. The matching between the first image and the second image may include spatial matching on a per-pixel level and / or global matching for the entire image.
[0060] During the operation of block 114, if the image statistics of the first image and the second image in the overlapping FOV area 64 are sufficiently different (e.g., if the difference is greater than a threshold), the ISP 80 may optionally fall back to image statistics from only one of the image sensors. Here, even if it has been determined that no image sensors are occluded, one of the image sensors may still output data that is clearly incorrect. For example, if the right image sensor 46-2 is somehow unable to focus, resulting in a lack of clarity and contrast in the second image, the ISP 80 may fall back to using only information from the first image and presenting that information to the user on both displays 14. As another example, even if no occlusion is detected, one image may still be significantly brighter than the other image based on the brightness histogram. In this case, priority may be given to the brighter image (because the brighter image tends to be easier to recover), so the ISP 80 may fall back to using only brightness information from the brighter image and applying it to both images.
[0061] During operation of block 116, if the image capture disparity detection block 90 detects an anomaly in a portion of at least one of the images, the ISP 80 may optionally exclude any image statistics from that portion of the image. For example, the ISP 80 may selectively exclude one or more regions of the image from a brightness histogram calculation or a color histogram calculation. Figure 5 The examples are merely illustrative. In general, blocks 110, 112, 114, and 116 may be executed in any desired order, or at least some of these blocks may be executed in parallel.
[0062] Figure 5 The operations described are exemplary. In some embodiments, one or more of the operations described may be modified, replaced, or omitted. In some embodiments, one or more of the operations described may be performed in parallel. In some embodiments, additional processes may be added or inserted between the operations described. If desired, the order of certain operations may be reversed or changed, and / or the timing of the operations described may be adjusted so that they occur at slightly different times. In some embodiments, the operations described may be distributed in a larger system.
[0063] Combination of the above Figures 1 to 5 The methods and operations described may be performed by the components of device 10 using software, firmware, and / or hardware (e.g., dedicated circuits or hardware). Software code for performing these operations may be stored on a non-transitory computer-readable storage medium (e.g., a tangible computer-readable storage medium) stored on one or more of the components of device 10 (e.g., a computer-readable storage medium). Figure 1 The software code may sometimes be referred to as software, data, instructions, program instructions, or code. Non-transitory computer-readable storage media may include a drive, non-volatile memory such as non-volatile random access memory (NVRAM), a removable flash drive or other removable media, other types of random access memory, etc. The software stored on the non-transitory computer-readable storage medium may be executed by processing circuitry on one or more of the components of the device 10 (e.g., one or more processors in the control circuitry 20). The processing circuitry may include a microprocessor, an application processor, a digital signal processor, a central processing unit (CPU), an application-specific integrated circuit having a processing circuit, or other processing circuitry.
[0064] There are many different types of electronic systems that can enable a person to sense and / or interact with various XR environments. Examples include head-mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers.
[0065] According to one embodiment, a method of operating an electronic device having at least a first image sensor and a second image sensor is provided, the method comprising: capturing a first image using the first image sensor having a first field of view; capturing a second image using the second image sensor having a second field of view different from the first field of view; determining whether the first image sensor is currently obscured; and synchronizing processing of the first image and the second image by processing the first image and the second image using information associated with the second image in response to determining that the first image sensor is currently obscured.
[0066] According to another embodiment, the method includes ignoring information associated with the first image captured using the first image sensor that is currently obscured.
[0067] According to another embodiment, the method includes: displaying the first processed image using a first display in the electronic device; and displaying the second processed image using a second display in the electronic device.
[0068] According to another embodiment, determining whether the first image sensor is currently obscured includes comparing thumbnail information associated with the first image with thumbnail information associated with the second image.
[0069] According to another embodiment, determining whether the first image sensor is currently obscured includes comparing brightness information associated with the first image with brightness information associated with the second image.
[0070] According to another embodiment, determining whether the first image sensor is currently obscured includes determining that a brightness histogram of the first image is skewed toward darker values relative to a brightness histogram of the second image.
[0071] According to another embodiment, determining whether the first image sensor is currently obscured includes comparing color information associated with the first image with color information associated with the second image.
[0072] According to another embodiment, determining whether the first image sensor is currently obscured includes comparing focus information associated with the first image with focus information associated with the second image.
[0073] According to another embodiment, determining whether the first image sensor is currently obscured includes comparing a first integration time associated with the first image to a second integration time associated with the second image.
[0074] According to another embodiment, the synchronization processing of the first image and the second image includes: synchronizing automatic exposure of the first image and the second image using brightness information associated with the second image.
[0075] According to another embodiment, the synchronization processing of the first image and the second image includes: using color information associated with the second image to synchronize automatic white balance of the first image and the second image.
[0076] According to another embodiment, the synchronization processing of the first image and the second image includes: synchronizing tone mapping of the first image and the second image using histogram information associated with the second image.
[0077] According to another embodiment, synchronizing the processing of the first image and the second image includes: using pixel information associated with the second image to synchronize one or more additional image signal processing functions of the first image and the second image.
[0078] According to another embodiment, the method includes, in response to determining that the first image sensor is currently obscured, outputting a notification indicating that the first image sensor is currently obscured.
[0079] According to another embodiment, the method includes: using a first display in the electronic device to display the first processed image; using a second display in the electronic device to display the second processed image; and in response to determining that the first image sensor is currently obscured, using a portion of the second processed image to fill a portion of the first processed image that is currently obscured on the first display.
[0080] According to another embodiment, the first field of view and the second field of view overlap in an overlapping field of view (FOV) region, and the method includes: processing the first image and the second image using one or more combined image signal processing control parameters calculated based on information associated with the first image within the overlapping FOV region and information associated with the second image within the overlapping FOV region.
[0081] According to another embodiment, the method includes: processing the first image and the second image using one or more combined image signal processing control parameters calculated based on information associated with the first image and the second image within a total field of view (FOV) area equal to the union of the first field of view and the second field of view.
[0082] According to another embodiment, the method includes excluding information from at least a portion of the first image or from at least a portion of the second image when processing the first image and the second image.
[0083] According to another embodiment, the method includes processing the first image and the second image using one or more spatial weight maps that assign varying weights to different parts of the images.
[0084] According to another embodiment, the one or more spatial weight maps assign greater weights to a central portion of the first image and the second image, to a portion of the first image and the second image aligned with a user's gaze point, or to a portion of the first image and the second image having a light source.
[0085] According to one embodiment, a method of operating an electronic device having at least a first image sensor and a second image sensor is provided, the method comprising: capturing a first image using the first image sensor having a first field of view; capturing a second image using the second image sensor having a second field of view different from the first field of view, the first field of view and the second field of view coinciding in an overlapping field of view (FOV) region; and processing the first image and the second image using one or more combined image signal processing control parameters calculated based on information associated with the first image within the overlapping FOV region and information associated with the second image within the overlapping FOV region.
[0086] According to another embodiment, processing the first image and the second image using the one or more combined image signal processing control parameters calculated based on information associated with the first image within the overlapping FOV area and information associated with the second image within the overlapping FOV area includes matching color, tone mapping, brightness, or noise between the first image and the second image.
[0087] According to another embodiment, matching color, tone mapping, brightness or noise between the first image and the second image includes performing spatial matching on a per-pixel level and performing global matching for the entire image.
[0088] According to another embodiment, the method includes: processing the first image and the second image using additional combined image signal processing control parameters calculated based on information associated with the first image within a total field of view (FOV) area equal to the union of the first field of view and the second field of view, and information associated with the second image within the total FOV area.
[0089] According to another embodiment, processing the first image and the second image using the additional combined image signal processing control parameters calculated based on information associated with the first image and the second image within the total FOV area includes estimating the type of light sources in the first image and the second image.
[0090] According to another embodiment, the estimated type of illuminant is a single illuminant used for global white balance correction of the first image and the second image.
[0091] According to another embodiment, the estimated type of illuminant includes a plurality of illuminants used for spatially varying white balance correction of the first image and the second image.
[0092] According to another embodiment, the method includes calculating a weighted sum of the information associated with the first image within the total FOV area and the information associated with the second image within the total FOV area.
[0093] According to another embodiment, the method includes calculating a weighted sum of the information associated with the first image within the overlapping FOV area and the information associated with the second image within the overlapping FOV area.
[0094] According to one embodiment, a device is provided, comprising: a first image sensor having a first field of view and configured to capture a first video feed; a second image sensor having a second field of view different from the first field of view and configured to capture a second video feed; an image statistics collection subsystem configured to collect first image statistics associated with the first video feed and to collect second image statistics associated with the second video feed; an image capture parallax detection subsystem configured to compare the first image statistics with the second image statistics; and a plurality of image signal processing blocks configured to synchronize processing of the first video feed and the second video feed based on the comparison of the first image statistics with the second image statistics.
[0095] The foregoing is merely illustrative and various modifications may be made to the embodiments described. The foregoing embodiments may be implemented independently or in any combination.
Claims
1. A method of operating an electronic device having at least a first image sensor and a second image sensor, the method comprising: capturing a first image using the first image sensor having a first field of view; capturing a second image using the second image sensor having a second field of view different from the first field of view; determining whether the first image sensor is currently blocked; as well as In response to determining that the first image sensor is currently obscured, processing of the first image and the second image is synchronized by processing the first image and the second image using information associated with the second image.
2. The method according to claim 1, further comprising: Information associated with the first image captured using the first image sensor that is currently obscured is ignored.
3. The method according to claim 1, further comprising: displaying a first processed image using a first display in the electronic device; as well as A second processed image is displayed using a second display in the electronic device.
4. The method of claim 1 , wherein determining whether the first image sensor is currently obscured comprises: Thumbnail information associated with the first image is compared with thumbnail information associated with the second image.
5. The method of claim 1 , wherein determining whether the first image sensor is currently obscured comprises: Luminance information associated with the first image and luminance information associated with the second image are compared, and wherein determining whether the first image sensor is currently obscured comprises determining that a luminance histogram of the first image is skewed toward darker values relative to a luminance histogram of the second image.
6. The method of claim 1 , wherein determining whether the first image sensor is currently obscured comprises: Color information associated with the first image is compared to color information associated with the second image.
7. The method of claim 1 , wherein determining whether the first image sensor is currently obscured comprises: Focus information associated with the first image is compared to focus information associated with the second image.
8. The method of claim 1 , wherein determining whether the first image sensor is currently obscured comprises: A first integration time associated with the first image and a second integration time associated with the second image are compared.
9. The method of claim 1, wherein synchronizing the processing of the first image and the second image comprises: Automatic exposure of the first image and the second image is synchronized using brightness information associated with the second image.
10. The method of claim 1, wherein synchronizing the processing of the first image and the second image comprises: Automatic white balance of the first image and the second image is synchronized using color information associated with the second image.
11. The method of claim 1 , wherein synchronizing the processing of the first image and the second image comprises: Tone mapping of the first image and the second image is synchronized using histogram information associated with the second image.
12. The method of claim 1, wherein synchronizing the processing of the first image and the second image comprises: One or more additional image signal processing functions of the first image and the second image are synchronized using pixel information associated with the second image.
13. The method according to claim 1, further comprising: In response to determining that the first image sensor is currently obscured, a notification is output indicating that the first image sensor is currently obscured.
14. The method according to claim 1, further comprising: displaying a first processed image using a first display in the electronic device; displaying a second processed image using a second display in the electronic device; as well as In response to determining that the first image sensor is currently obscured, a portion of the second processed image is used to fill a portion of the first processed image that is currently obscured on the first display.
15. The method of claim 1, wherein the first field of view and the second field of view coincide in an overlapping field of view (FOV) region, the method further comprising: The first image and the second image are processed using one or more combined image signal processing control parameters calculated based on information associated with the first image within the overlapping FOV area and information associated with the second image within the overlapping FOV area.
16. The method according to claim 1, further comprising: The first and second images are processed using one or more combined image signal processing control parameters calculated based on information associated with the first and second images within a total field of view (FOV) area equal to the union of the first and second fields of view.
17. The method according to claim 1, further comprising: When processing the first image and the second image, information is excluded from at least a portion of the first image or from at least a portion of the second image.
18. The method according to claim 1, further comprising: The first image and the second image are processed using one or more spatial weight maps, which assign varying weights to different parts of the images, wherein the one or more spatial weight maps assign greater weights to a central part of the first image and the second image, to a part of the first image and the second image that is aligned with a user's gaze point, or to a part of the first image and the second image that has a light source.
19. A method of operating an electronic device having at least a first image sensor and a second image sensor, the method comprising: capturing a first image using the first image sensor having a first field of view; capturing a second image using the second image sensor having a second field of view different from the first field of view, wherein the first field of view and the second field of view coincide in an overlapping field of view (FOV) region; as well as The first image and the second image are processed using one or more combined image signal processing control parameters calculated based on information associated with the first image within the overlapping FOV area and information associated with the second image within the overlapping FOV area.
20. An apparatus comprising: a first image sensor having a first field of view and configured to capture a first video feed; a second image sensor having a second field of view different from the first field of view and configured to capture a second video feed; an image statistics collection subsystem configured to collect first image statistics associated with the first video feed and to collect second image statistics associated with the second video feed; an image capture disparity detection subsystem configured to compare the first image statistics to the second image statistics; and A plurality of image signal processing blocks are configured to synchronize processing of the first video feed and the second video feed based on the comparison of the first image statistics and the second image statistics.
