Stereoscopic capture using cameras with different fields of view

Abstract

An electronic device is provided comprising a first camera having a first field of view configured to capture a first image, a second camera having a second field of view configured to capture a second image, and a control circuit configured to output stereoscopic content based on the first and second images. The control circuit may include image signal processors configured to process the first and second images, additional processors for performing stereo correction and image stabilization on the first and second images, and a compression block for combining images output from the additional processors.

Description

Technology Field

[0001] This application claims priority to U.S. Patent Application No. 18 / 645,357 filed on April 24, 2024, which claims the benefit of U.S. Provisional Application No. 63 / 505,350 filed on May 31, 2023, the entirety of which is incorporated herein by reference.

[0002] Technology field

[0003] The present invention generally relates to electronic devices, and more specifically to electronic devices having cameras. Background Technology

[0004] Electronic devices may include one or more cameras for capturing an image or video feed of a scene. The electronic device may include a wide camera having a first field of view, an ultrawide camera having a second field of view larger than the first field of view, and a telephoto camera having a third field of view smaller than the first field of view.

[0005] The electronic device may include one or more cameras for capturing image or video feeds of a real-world environment. The electronic device may include a wide image sensor configured to capture corresponding wide images, an ultra-wide image sensor configured to capture corresponding ultra-wide images, and a control circuit for processing and combining the wide images and ultra-wide images to generate stereoscopic images.

[0006] One aspect of the present disclosure provides a method for operating an electronic device, comprising the steps of: capturing a first image using a first image sensor having a first field of view; capturing a second image using a second image sensor having a second field of view different from the first field of view; and outputting stereoscopic content based on the first image captured using the first image sensor having the first field of view and the second image captured using the second image sensor. The method may include the steps of: processing the first image using a first image signal processor to output a first processed image; and processing the second image using a second image signal processor to output a second processed image. The method may include the steps of: performing stereo rectification and image stabilization operations on the first and second processed images to output corresponding first and second rectified and stabilized images; and compressing the first and second rectified and stabilized images using a multi-view video encoding method to generate a stereoscopic video stream. The present method may include the step of acquiring stabilization information for a first image sensor, and the step of synchronizing image stabilization between the first and second image sensors by applying the stabilization information for the first image sensor to a second processed image captured using a second image sensor. The present method may include the step of acquiring a rotation matrix associated with the first image sensor, the step of calculating a corrected and stabilized pose based on the rotation matrix associated with the first image sensor and motion data, the step of correcting the pose of the first image sensor, and the step of stabilizing the corrected pose of the first image sensor using a time filter.

[0007] The present method may further include the step of calculating a first homography based on a corrected and stabilized pose, motion data associated with a first image sensor, intrinsic data associated with the first image sensor, and intrinsic data associated with a target image sensor, and the step of calculating a second homography based on a corrected and stabilized pose, motion data associated with a second image sensor, intrinsic data associated with the second image sensor, and intrinsic data associated with a target image sensor. The present method may further include the step of generating a first corrected and stabilized image by warping a first processed image using the first homography, and the step of generating a second corrected and stabilized image by warping a second processed image using the second homography. The present method may further include the step of generating a still stereoscopic pair.

[0008] One aspect of the present disclosure provides an electronic device comprising: a first camera configured to capture a first image having a first field of view; a second camera configured to capture a second image having a second field of view different from the first field of view; and a control circuit configured to output stereoscopic content based on a first image captured from a first camera having a first field of view and a second image from a second camera having a second field of view different from the first field of view. The control circuit may include: a first image signal processor configured to receive a first image captured by a first camera and output a corresponding first processed image; a second image signal processor configured to receive a second image captured by a second camera and output a corresponding second processed image; a first additional processor configured to receive a first processed image, motion data associated with the first camera, and first calibration data, and further configured to generate a first corrected and stabilized image; a second additional processor configured to receive a second processed image, motion data associated with the second camera, and second calibration data, and further configured to generate a second corrected and stabilized image; and a video compression block configured to receive the first and second corrected and stabilized images and generate a corresponding stereoscopic video stream. The electronic device may further include a circuit for outputting still images.

[0009] One aspect of the present disclosure provides a method for operating an electronic device, comprising the steps of: capturing a first image using a first camera having a first field of view; capturing a second image using a second camera having a second field of view different from the first field of view; generating stereoscopic content based on the first and second captured images; and outputting a user notification to improve the quality of the stereoscopic content. The method may include the step of detecting an incorrect stereoscopic capture orientation of the electronic device and then outputting a user notification to switch to a correct stereoscopic capture orientation. The method may include the step of detecting whether the first camera is obscured and then outputting a user notification that the first camera is obscured. The method may include the step of detecting lighting conditions of the first or second image and then outputting a user notification that the lighting conditions are below a threshold. The method may include the step of detecting an external object within the field of view of the first camera and then outputting a user notification that the external object is within the field of view of the first camera. The present method may include the step of detecting motion or jitter of the first and second cameras and then outputting a user notification to stop moving or to keep them stationary. Brief explanation of the drawing

[0010] FIG. 1 is a schematic diagram of an exemplary electronic device according to some embodiments. FIG. 2 is a front perspective view of an electronic device of the type shown in FIG. 1 according to some embodiments. FIG. 3 is a rear perspective view of an electronic device of the type shown in FIG. 1 and FIG. 2 according to some embodiments. FIG. 4 is a drawing of an exemplary electronic device having image sensors of different focal lengths used to generate a stereoscopic video stream according to some embodiments. FIG. 5 is a diagram of exemplary operations for warping images from two image sensors according to some embodiments. FIG. 6 is a drawing illustrating additional subsystems configured to generate still stereoscopic images according to some embodiments. FIG. 7 is a drawing illustrating additional subsystems coupled to image signal processors and configured to generate still stereoscopic images according to some embodiments. FIG. 8 is a flowchart of exemplary steps for operating an electronic device of the type illustrated in FIG. 1 to 7 according to some embodiments. Specific details for implementing the invention

[0011] An electronic device comprising a plurality of cameras configured to capture stereoscopic video and / or images is provided. For example, a wide camera and an ultra-wide camera on a cellular phone may be used to capture stereoscopic content. A schematic diagram of an exemplary electronic device (10) configured to capture stereoscopic content is shown in FIG. 1. The device (10) of FIG. 1 may operate as a standalone device and / or the resources of the device (10) may be used to communicate with external electronic equipment. For example, a communication circuit within the device (10) may be used to transmit user input information, sensor information, and / or other information to external electronic devices (e.g., wirelessly or via a wired connection) and / or to receive such information from external electronic devices. Each of these external devices may include components of the type illustrated by the device (10) of FIG. 1.

[0012] As illustrated in FIG. 1, the electronic device (10) may include a control circuit (14). The control circuit (14) may include a storage such as a storage circuit (16). The storage circuit (16) may include a hard disk drive storage, non-volatile memory (e.g., flash memory, or other electrically programmable read-only memory configured to form a solid-state drive), volatile memory (e.g., static or dynamic random access memory), etc. The storage circuit (16) may include a storage integrated into the device (10) and / or a removable storage medium.

[0013] The control circuit (14) may include a processing circuit such as the processing circuit (18). The processing circuit (18) may be used to control the operation of the device (10). The processing circuit (18) may include one or more microprocessors, microcontrollers, digital signal processors, host processors, baseband processor integrated circuits, application integrated circuits, central processing units (CPUs), power management units, audio chips, etc. The control circuit (14) may be configured to perform operations on the device (10) using hardware (e.g., dedicated hardware or circuits), firmware, and / or software. Software code for performing operations on the device (10) may be stored on the storage circuit (16) (e.g., the storage circuit (16) may include non-transient (tangible) computer-readable storage media for storing software code). Software code may sometimes be referred to as program instructions, software, data, instructions, or code. The software code stored on the storage circuit (16) can be executed by the processing circuit (18).

[0014] The control circuit (14) can be used to execute software on the device (10), such as satellite navigation applications, internet browsing applications, VoIP (voice-over-internet-protocol) phone call applications, email applications, media playback applications, operating system functions, etc. To support interactions with external equipment, the control circuit (14) can be used to implement communication protocols. Communication protocols that can be implemented using the control circuit (14) include internet protocols, wireless local area network (WLAN) protocols (e.g., IEEE 802.11 protocols - sometimes referred to as Wi-Fi®), Bluetooth® protocols or other short-range wireless communication link protocols such as wireless personal area network (WPAN) protocols, IEEE 802.11ad protocols (e.g., ultra-wideband protocols), cellular phone protocols (e.g., 3G protocols, 4G (LTE) protocols, 5G protocols, etc.), antenna diversity protocols, satellite navigation system protocols (e.g., global positioning system (GPS) protocols, global navigation satellite system (GLONASS) protocols, etc.), antenna-based spatial ranging protocols (e.g., radio detection and ranging (RADAR) protocols or Includes other desired range detection protocols for signals transmitted at millimeter and centimeter frequency ranges), or any other desired communication protocols. Each communication protocol may be associated with a corresponding Radio Access Technology (RAT) that specifies the physical connection methodology used to implement the protocol.

[0015] To support communications between the device (10) and external equipment, the control circuit (14) may communicate using the communication circuit (20). The communication circuit (20) may include an antenna, a radio frequency transceiver circuit, and other wireless communication circuits and / or wired communication circuits. The communication circuit (20), which may sometimes be referred to as the control circuit and / or control and communication circuit, may support bidirectional wireless communication between the device (10) and external equipment (e.g., companion devices such as a computer, cellular phone, or other electronic device, point devices, computer styluses, or other input devices, speakers, or other output devices, accessories, etc.) via a wireless link.

[0016] The communication circuit (20) can transmit and / or receive radio frequency signals within a corresponding frequency band (sometimes referred to in this specification as a communication band or simply "band") at radio frequencies. The frequency bands processed by the communication circuit section (20) are wireless local area network (WLAN) frequency bands (e.g., Wi-Fi® (IEEE 802.11) or other WLAN communication bands) such as 2.4 GHz WLAN band (e.g., 2400 to 2480 MHz), 5 GHz WLAN band (e.g., 5180 to 5825 MHz), Wi-Fi® 6E band (e.g., 5925 to 7125 MHz) and / or other Wi-Fi® bands (e.g., 1875 to 5160 MHz), wireless personal area network (WPAN) frequency bands such as 2.4 GHz Bluetooth® band or other WPAN communication bands, cellular phone frequency bands (e.g., bands of about 600 MHz to about 5 GHz, 3G bands, 4G LTE bands, 5G New Radio Frequency Range 1 (FR1) bands below 10 GHz, 5G New Radio Frequency Range 2 (FR2) bands between 20 and 60 GHz, etc.), other centimeter wave or millimeter wave frequency bands between 10 and 300 GHz, near-field communication frequency bands (e.g., 13.56 MHz), satellite navigation frequency bands (e.g., GPS band between 1565 and 1610 MHz, Global Navigation Satellite System (GLONASS) band, BeiDou Navigation Satellite System (BDS) band, etc.), ultra-wideband (UWB) frequency bands operating under the IEEE 802.15.4 protocol and / or other ultra-wideband communication protocols, communication bands under the 3GPP radio communication standards family, communication bands under the IEEE 802.XX standards family, and / or any It can include other desired frequency bands of interest.

[0017] If desired, the device (10) may include a power circuit for transmitting and / or receiving wired and / or wireless power and may include a battery or other energy storage device. For example, the device (10) may include a wireless power coil and a rectifier for receiving wireless power provided to another circuit part within the device (10).

[0018] The device (10) may include input / output devices such as input / output devices (22). Electronic components such as input / output devices (22) may be used to collect user input, to collect information about the environment surrounding the user, and / or to provide output to the user. The input / output devices (22) may include one or more displays such as a display (24). The display (24) may include one or more display devices such as an organic light-emitting diode display panel (a panel in which organic light-emitting diode pixels are formed on a polymer substrate or silicon substrate including a pixel control circuit), a liquid crystal display panel, a microelectromechanical system display (e.g., a two-dimensional mirror array or a scanning mirror display device), a display panel having a pixel array formed of a crystalline semiconductor light-emitting diode die (sometimes called a microLED), and / or other display devices.

[0019] The input / output devices (22) may also include sensors (26). The sensors (26) within the input / output devices (22) may include image sensors (e.g., visible light cameras, infrared cameras, cameras sensitive to multiple wavelengths, depth sensors, structured light sensors and / or three-dimensional camera systems such as depth sensors based on stereo imaging devices that capture three-dimensional images, time-of-flight cameras, etc.), force sensors (e.g., strain gauges, capacitive force sensors, resistive force sensors, etc.), audio sensors such as microphones, touch and / or proximity sensors such as capacitive sensors such as touch sensors forming buttons, trackpads, or other input devices, and other sensors. If desired, the sensors (26) may be optical sensors, such as optical sensors that emit and detect light, ultrasonic sensors, optical touch sensors, optical proximity sensors, and / or other touch sensors and / or proximity sensors, monochromatic and color ambient light sensors, fingerprint sensors, iris scanning sensors, retinal scanning sensors, and other biometric sensors, temperature sensors, sensors for measuring three-dimensional non-contact gestures ("air gestures"), pressure sensors, sensors for detecting position, orientation and / or motion (e.g., accelerometers, magnetic sensors, such as compass sensors, gyroscopes, and / or inertial measurement units including some or all of these sensors), health sensors such as blood oxygen sensors, heart rate sensors, blood flow sensors, and / or other health sensors, radio frequency sensors, optical sensors, such as self-mixing sensors and light detection and ranging (lidar) sensors, humidity sensors, It may include moisture sensors, eye tracking sensors, electromyography sensors for detecting muscle activity, face sensors, interferometer sensors, time-of-flight sensors, magnetic sensors, resistive sensors, distance sensors, angle sensors, and / or other sensors.

[0020] In some arrangements, the device (10) may use sensors (26) and / or other input / output devices (22) to collect user input. For example, input / output devices (22), such as buttons, may be used to collect button press input, touch sensors overlapping with displays may be used to collect user touch screen input, touchpads may be used to collect touch input, microphones may be used to collect audio input (e.g., voice commands), and accelerometers may be used to monitor when a finger contacts an input surface and accordingly to collect finger press input.

[0021] Input / output devices (22) may include optical components, such as depth sensors (e.g., structured light sensors or other sensors that collect three-dimensional image data), optical proximity sensors, ambient light sensors (e.g., color ambient light sensors), optical time-of-flight sensors, and other sensors (16) that are sensitive to visible light and / or infrared light and can emit visible light and / or infrared light (e.g., devices (22) may include optical sensors that emit and / or detect light). For example, a visible light image sensor in a camera may have a visible light flash or an associated infrared flood illuminator to provide illumination while the image sensor captures two-dimensional and / or three-dimensional images. An infrared camera, such as an infrared structured light camera that captures three-dimensional infrared images, may have an infrared flood illuminator that emits infrared flood illumination and / or may have a dot projector that emits an array of infrared light beams. Infrared proximity sensors emit infrared light and can detect infrared light after it is reflected from a target object.

[0022] If desired, the electronic device (10) may include additional components (e.g., see other devices (28) within the input / output devices (22)). Additional components may include haptic output devices, actuators for moving movable structures within the device (10), audio output devices such as speakers, light sources such as light-emitting diodes for status indicators, light-emitting diodes for illuminating parts of the housing and / or display structure, other optical output devices, and / or other circuitry for collecting input and / or providing output. The device (10) may also include a battery or other energy storage device, a connector port for supporting wired communication with auxiliary equipment and receiving wired power, and other circuitry.

[0023] FIG. 2 is a front perspective view of an electronic device (10) of the type illustrated in FIG. 1. The electronic device (10) may be a computing device such as a laptop computer, a computer monitor including an embedded computer, a tablet computer, a cellular phone, a media player, or other handheld or portable electronic device, a smaller device such as a wristwatch device (e.g., a watch with a wrist strap), a pendant device, a headphone or earpiece device, a device embedded in glasses or other equipment worn on a user's head, or other wearable or small device, a television, a computer display not including an embedded computer, a gaming device, a navigation device, an embedded system such as a system in which electronic equipment with a display is mounted in a kiosk or a vehicle, equipment implementing the functions of two or more of these devices, or other electronic equipment. In the exemplary configuration of FIG. 2, the device (10) is a portable device such as a cellular phone, a media player, a tablet computer, a wrist device, or other portable computing device. If desired, other configurations for the device (10) may be used. The example in FIG. 2 is merely illustrative.

[0024] In the example of FIG. 2, the device (10) includes a display such as a display (14) mounted within a housing (12). In particular, the display (14) (or a transparent cover layer covering the display (14)) may be mounted within the housing (12) and may form at least a portion of the front surface of the device (10). Although not shown in FIG. 2, the device (10) may also have an opposing rear surface formed by the housing (12). The housing (12), which may sometimes be referred to as an enclosure or a case, may be formed from plastic, glass, ceramic, fiber composite, metal (e.g., stainless steel, aluminum, titanium, gold, etc.), other suitable materials, or any combination of two or more of these materials. The housing (12) may be formed using an integral configuration in which part or all of the housing (12) is machined or molded as a single structure, or it may be formed using a plurality of structures (e.g., an internal frame structure, one or more structures forming the external housing surfaces, etc.).

[0025] The display (14) may be a touchscreen display incorporating a layer of conductive capacitive touch sensor electrodes or other touch sensor components (e.g., resistive touch sensor components, acoustic touch sensor components, force-based touch sensor components, light-based touch sensor components, etc.), or it may be a non-touch-sensitive display. Capacitive touchscreen electrodes may be formed from an array of indium tin oxide pads or other transparent conductive structures. The display (14) may include an array of pixels formed from liquid crystal display (LCD) components, an array of electrophoretic pixels, an array of plasma pixels, an array of organic light-emitting diode pixels or other light-emitting diodes, an array of electrowetting pixels, or pixels based on other display technologies. The display cover layer for the display (14) may be flat or curved and may have a rectangular contour, a circular contour, or a contour of other shapes. If desired, openings may be formed in the display cover layer. For example, openings may be formed in the display cover layer to accommodate buttons, speaker ports, sensors, or other components. For example, openings may be formed in the housing (12) to form communication ports (e.g., audio jack ports, digital data ports, etc.), to form openings for buttons, or to form audio ports (e.g., openings for speakers and / or microphones).

[0026] FIG. 3 is a rear perspective view of an electronic device (10) of the type illustrated in FIG. 1 and FIG. 2. As illustrated in FIG. 3, a plurality of cameras, including a first image sensor (30-1) and a second image sensor (30-2), may be placed on the rear surface (R) of the device (10). Thus, the image sensors (30-1, 30-2) are sometimes referred to as rear-facing cameras. Both of the image sensors (30-1, 30-2) may be color image sensors (e.g., cameras configured to capture color images). The image sensors (30-1, 30-2) (and corresponding lenses) may be configured with different fields of view (FoV). For example, the image sensor (30-1) may have a first FoV equivalent to a first focal length, while the image sensor (30-2) may have a second FoV equivalent to a second focal length smaller than the first focal length (e.g., the second FoV of the camera (30-2) may also be larger than the first FoV of the camera (30-1)). In some embodiments, the first camera (30-1) may be referred to as a wide ("W") image sensor, while the second camera (30-2), having a much wider field of view (and a shorter focal length), may be referred to as an ultrawide ("UW") image sensor.

[0027] The electronic device (10) may have a rectangular shape having elongated longitudinal dimensions along the longitudinal axis (32). Image sensors (30-1, 30-2) may be placed at different points along the longitudinal axis (32). When the device (10) is held upright (e.g., when the longitudinal axis (32) of the device (10) is oriented orthogonal to the ground), the device (10) may have the image sensors (30-1, 30-2) positioned above or above each other. verticallyIt may be referred to as operating in a "portrait" orientation so as to be positioned. When the device (10) is held sideways (e.g., when the longitudinal axis (32) of the device (10) is oriented parallel to the ground), the device (10) is positioned such that the image sensors (30-1, 30-2) are positioned laterally or horizontally It can be referred to as operating in a "landscape" orientation so as to be positioned.

[0028] The example of FIG. 3, in which the device (10) includes two rear-facing cameras, is merely exemplary. Generally, the device (10) may include only one rear-facing camera, two or more rear-facing cameras, three or more rear-facing cameras, four or more rear-facing cameras, four to ten rear-facing cameras, or more than ten rear-facing cameras, each having the same or different field of view. If desired, the image sensors (30-1, 30-2) may alternatively be positioned along different points on the latitude axis perpendicular to the longitudinal axis (32) across the rear surface (R) of the device (10). In certain embodiments, the device (10) may also include a plurality of cameras having the same or different fields of view and focal lengths on the front surface of the device (e.g., see FIG. 2). Configurations comprising a first wide image sensor (30-1) and a second ultra-wide image sensor (30-2) configured in a manner at least illustrated in FIG. 3 are sometimes described as examples in this specification.

[0029] According to some embodiments, a plurality of cameras on the device (10) may be employed to capture stereoscopic content. Stereoscopic content may refer to visual media (e.g., videos or still images) that have a sense of depth and dimension by presenting two slightly different perspectives of the same scene to the user's eyes. FIG. 4 is a drawing illustrating hardware and / or software subsystems within the device (10) that may be used to capture a stereoscopic video stream. As illustrated in FIG. 4, the device (10) may include image signal processors (ISPs) such as a first image sensor (30-1), a second image sensor (30-2), a first image signal processor (50-1) and a second image signal processor (50-2), a first processor downstream of and associated with the ISP (50-1) such as a first processor (60-1), a second processor downstream of and associated with the ISP (50-2) such as a second processor (60-2), a codec (coder decoder) block such as a codec (70), memory devices such as a memory (40), and sensors such as a motion sensor (42).

[0030] The first image sensor (30-1) (Camera A) may be a "wide" camera having a first field of view and a first focal length. The second image sensor (30-2) (Camera B) may be an "ultra-wide" camera having a second field of view wider than the first field of view and a second focal length shorter than the first focal length. This example, in which the image sensors (30-1, 30-2) are wide and ultra-wide cameras, respectively, is exemplary and is not intended to limit the scope of the embodiments. Generally, two or more image sensors (30) having the same or different fields of view may be used to generate stereoscopic content using the techniques described herein. The image sensor (30-1) may output a first raw (unprocessed) image to an image signal processor (50-1), while the image sensor (30-2) may output a second raw (unprocessed) image to an image signal processor (50-2).

[0031] Each image signal processor (50) (e.g., ISPs (50-1, 50-2)) may be configured to perform classical image signal processing functions that rely solely on the input of the live camera feed itself. For example, each ISP block (50) may be configured to perform automatic exposure (AE), automatic color correction (sometimes referred to as automatic white balancing), tone mapping (e.g., global and / or local tone mapping), gamma correction, shade correction, noise reduction, black level adjustment, demosaicing, image sharpening, high dynamic range (HDR) correction, color space conversion, and / or other image signal processing functions (only a few are named). In the example of FIG. 4, each ISP block (50) may be further configured to crop or downscale raw images received from image sensors. For example, an image signal processor (50-1) may receive a first raw image from an image sensor (30-1), perform one or more of the aforementioned ISP functions to obtain a first processed image, and crop or downscale the first processed image using an internal cropping / scaling subblock (52). Similarly, an image signal processor (50-2) may receive a second raw image from an image sensor (30-2), perform one or more of the aforementioned ISP functions to obtain a second processed image, and crop or downscale the second processed image using an internal cropping / scaling subblock (52).

[0032] A first processed image, sometimes referred to herein as a first scaled and processed image, may be transmitted to a first downstream processor (60-1). A second processed image, sometimes referred to herein as a second scaled and processed image, may be transmitted to a second downstream processor (60-2). The first and second processed images may have different resolutions and different fields of view. The processors (60-1, 60-2) may each be implemented as a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a digital signal processor (DSP), a field programmable gate array (FPGA), and / or other types of processors.

[0033] In order for the first and second processed images output from the ISP blocks (50-1, 50-2) to become stereoscopic pairs, the processors (60-1, 60-2) may be configured to perform stereo correction operations and image stabilization operations on the processed images. As illustrated in FIG. 4, the processors (60-1, 60-2) may each include subblocks (64, 66) for performing stereo correction and stabilization operations. In one example, the stereo correction operations may be performed before the image stabilization operations. In another example, the stereo correction operations may be performed after the image stabilization operations. In yet another example, the stereo correction operations may be performed in parallel (simultaneously) with the image stabilization operations.

[0034] Stereo correction may refer to a process of transforming a pair of stereo images or videos so that corresponding points within two views lie on the same horizontal scan line. Stereo correction operations or functions performed by the subblocks (64) may produce epipolar alignment (e.g., so that epipolar lines are horizontal). Stereo correction (sometimes referred to as "stereo alignment") may refer to a process of aligning a pair of images or videos so that corresponding points within two views lie on the same horizontal scan line. Stereo alignment may be achieved by finding translation and / or rotation between two views and then using this information to warp both images.

[0035] To perform stereo calibration, processors (60-1, 60-2) may receive stereo calibration calibration data stored in memory (40). Memory (40) may be non-volatile memory, volatile memory, and / or other types of storage that are part of the circuit portion (16) of FIG. 1. The stereo calibration calibration data may include extrinsic calibration data of the image sensor (30-1) and / or image sensor (30-2). "Extrinsic" calibration data may be referred to and defined herein as data associated with the 6 degrees of freedom (6DOF) of the image sensor in 3D space. For example, extrinsic calibration data of the image sensor rotation Orientation (e.g., pitch, roll, and yaw) and image sensor translation It may include information related to positioning (e.g., forward / rear displacement, up / down displacement, and left / right displacement).

[0036] In contrast, "intrinsic" data may be referred to and defined herein as data related to the way world coordinates are projected onto an image sensor (e.g., how 3D data is converted into 2D data). For example, intrinsic camera data may include the focal length of the image sensor, the optical center of the image sensor, the skew associated with the image sensor, and / or other intrinsic metrics that may potentially vary per frame. If desired, intrinsic data related to the image sensors (30-1 and / or 30-2) may also be stored in memory (40). In some embodiments, memory (40) contains extrinsic data associated with the image sensors (30-1) and / or image sensors (30-2). Calibration The inherent of both the data (e.g., stereo calibration data) and the image sensors (30-1, 30-2) Calibration Sensor data can be stored and provided to processors (60-1, 60-2).

[0037] Regarding image stabilization, stabilization (and pose) information associated with the first image sensor (30-1) can be used to stabilize an image output from the image sensor (30-2) to ensure that stabilization between the two cameras is synchronized. As illustrated by the dotted line (68), stabilization information for the first image sensor (30-1) can be applied to a second processed image captured using the second image sensor (30-2). Here, the stabilization / pose information associated with the sensor (30-1) can be applied to the output of the sensor (30-2). This is exemplary. As another example, stabilization / pose information from the ultra-wide sensor (30-2) can alternatively be applied to the output of the sensor (30-1). As another example, each image sensor (30) can rely on its own image stabilization / pose information. Device configurations in which the stabilization / pose information associated with the image sensor (30-1) is applied to the output of the image sensor (30-2) are sometimes described in this specification as examples.

[0038] Pose information used in image stabilization algorithms in subblocks (66) can be obtained using one or more motion sensors (42). The motion sensor(s) (42) can output motion data associated with the image sensor (30-1 and / or 30-2). The motion sensor(s) (42) can be considered as part of the sensors (26) of FIG. 1. As an example, the motion sensor(s) (42) may include visual inertial odometry (VIO) sensors for collecting information used to track the orientation and position of the device (10). The VIO sensors may include inertial measurement units (e.g., gyroscope, gyrocompass, accelerometer, magnetometer and / or other inertial sensors), one or more tracking cameras, and / or other position and motion sensors. The motion sensor (42) can directly determine the pose, movement, yaw, pitch, roll, etc. of the image sensor.

[0039] The motion sensor (42) can also be used to determine the current orientation and position of the device (10) within the environment. Thus, the sensors (42) are sometimes also referred to as position sensors. For example, the first motion sensor (42) can provide motion data associated with the first image sensor (30-1) to the stabilization subblock (66) in the processor (60-1), whereas the second motion sensor (42) can provide motion data associated with the second image sensor (30-2) to the stabilization subblock (66) in the processor (60-2). Motion data output by these types of motion sensors (42) can also be considered as "external" (non-calibrated) camera data.

[0040] Processors (60-1, 60-2) configured to perform stereo correction and image stabilization operations in this manner are sometimes referred to as stereo correction (alignment) and stabilization processors. Processor (60-1) can output a first corrected and stabilized image (labeled as Image_out1 in FIG. 4) to the first input of the codec block (70). Processor (60-2) can output a second corrected and stabilized image (labeled as Image_out2) to the second input of the codec block (70). Unlike the first and second processed images received at the inputs of the processors (60-1, 60-2) which may have different resolutions and different fields of view, the first and second corrected (aligned) and stabilized images generated at the outputs of the processors (60-1, 60-2) may have the same resolution and the same field of view (e.g., images provided at the inputs of the codec block (70) may have the same focal length without any camera or lens distortion).

[0041] The codec block (70) may be configured to implement, for example, MV-HEVC (Multiview High Efficiency Video Coding), which is a video compression protocol designed to provide efficient data compression for multi-view video content. Accordingly, the codec block (70) can generate an MV-HEVC video stream based on a series of first and second corrected and stabilized images received from processors (60-1, 60-2). MV-HEVC is an extension of the HEVC standard that supports encoding multiple views of a captured scene into a single data stream. MV-HEVC achieves efficient compression by utilizing similarities between different views of a scene. For example, an interview prediction scheme can allow motion and texture information from one view to predict motion and texture from another view. This prediction scheme can reduce data redundancy and improve compression efficiency. Accordingly, the codec block (70) is sometimes referred to as a video compression block.

[0042] This example in which the codec block (70) implements MV-HEVC is exemplary. If desired, the codec (70) may be configured to implement other types of multi-view encoding schemes to generate a stereoscopic video stream. The stereoscopic video stream may optionally be stored in the cloud within a remote or local database and may be played on a device having one or more displays capable of presenting stereoscopic video content. As an example, the stereoscopic video stream may be played on a head-mounted device having one or more displays to create a sense of depth and a 3D experience for the user. Various blocks illustrated in FIG. 4, such as blocks (50-1, 50-2, 60-1, 60-2, 70), may be collectively referred to as a control circuit (e.g., see control circuit (14) in FIG. 1).

[0043] The device (10) (e.g., a cellular phone) and the head-mounted device may have different viewing conditions. For example, the device (10) may operate under non-immersive viewing conditions, while the head-mounted device may operate under immersive viewing conditions that tend to be much darker. A stereoscopic video stream captured by the device (10) may be provided with metadata containing a chromatic adaptation matrix that is adapted to non-immersive viewing conditions. Because the viewing conditions of the head-mounted device are different from the viewing conditions of the device (10), the head-mounted device may optionally color correct the stereoscopic content captured using the device (10) using a modified version of the chromatic adaptation matrix included in the metadata. When the stereoscopic content captured using the device (10) is displayed on another device under non-immersive viewing conditions, the device may simply apply the same chromatic adaptation matrix included in the metadata to the stereoscopic content being displayed.

[0044] FIG. 5 is a diagram illustrating exemplary operations that can be performed using stereo correction (alignment) and image stabilization processors (60-1, 60-2). In the operations of block (80), a stereo correction 3D rotation matrix may be calculated for the image sensor (30-1). The stereo correction 3D rotation matrix may be a fixed matrix calculated based on extrinsic calibration data associated with the image sensor (30-1) and / or the image sensor (30-2). In the operations of block (82), a corrected and stabilized pose may be calculated by (1) correcting the pose from the image sensor (30-1) and then (2) stabilizing the corrected pose of the image sensor (30-1) (e.g., using a time filter). This corrected and stabilized pose (in this specification, "R T(defined as) can be calculated on a frame-by-frame basis. The corrected and stabilized pose is based on the rotation matrix calculated from the block (80) and provided by the motion sensor(s) (42), as in “R in this specification”. A It can be calculated based on rotation data associated with an image sensor (30-1), defined as and optionally expressed in world coordinates. The operations of blocks (80, 82) can be performed by a processor (60-1).

[0045] During the operations of block (84-1), a first stabilization homography (H1) can be calculated for the image sensor (30-1). Homography generally refers to a mathematical transformation that maps points within one plane to corresponding points within another plane. Homography can be used to describe the relationship between two images of the same scene taken from different angles or perspectives. Homography can sometimes be expressed as a matrix used to project points within one image to corresponding points within another image based on the intrinsic and extrinsic parameters of the image sensor.

[0046] In the example of FIG. 5, the first stabilization homography (H1) is the corrected and stabilized pose (R) calculated from the block (82). T ), as provided by the motion sensor(s) (42) (in this specification "R A Rotation or motion data associated with an image sensor (30-1) defined as "K", provided from memory (40) (in this specification "K" A Inherent calibration sensor data associated with an image sensor (30-1) defined as "K", and provided from memory (40) (in this specification "K" T It can be calculated based on intrinsic sensor data associated with a target image sensor (defined as and sometimes referred to as intrinsic target sensor data). Intrinsic calibration sensor data (K A) may include, for example, the focal length and / or optical center point of the image sensor (30-1). Intrinsic target sensor data (K T ) is the intrinsic calibration sensor data (K) of the sensor (30-1). A It may be the same as or different from ). For example, the optical center of the target camera may be dynamically set to be the same as the optical center of the image sensor (30-1), or statically set to the center of the image. The first stabilization homography (H1) can be calculated as follows:

[0047] H1= K A R A (R T ) -1 (K T ) -1 (1)

[0048] The first stabilization homography (H1) calculated from block (84-1) can be used as a warping mesh to perform the first warping function in block (86-1). The warping function may be, for example, a bicubic warping function for interpolating between related pixels. The warping operation (86-1) may take Image_in1 (e.g., a processed image received from the output of the image signal processor (50-1)) as input and then warp Image_in1 using the homography (H1) to generate a corresponding first corrected and stabilized image Image_out1. When configured in this way, correction (alignment) and stabilization are applied together in a single operation to warp the image. The operations of blocks (84-1, 86-1) may be performed by the processor (60-1).

[0049] The second stabilization homography (H2) is a corrected and stabilized pose (R) calculated from the block (82). T ), as provided by the motion sensor(s) (42) (in this specification "R BRotation data associated with an image sensor (30-2) defined as "K", provided from memory (40) (in this specification "K" B Implicit calibration sensor data associated with an image sensor (30-2) defined as ", and implicit sensor data (K) associated with a target image sensor as provided from memory (40) T It can be calculated based on ). Implicit calibration sensor data (K B ) may include, for example, the focal length and / or optical center point of the image sensor (30-2). The second stabilization homography (H2) can be calculated as follows:

[0050] H2= K B R B (R T ) -1 (K T ) -1 (2)

[0051] The second stabilization homography (H2) calculated from block (84-2) can be used as a warping mesh to perform the second warping function in block (86-2). The warping function may be, for example, a bicubic warping function for interpolating between related pixels. The warping operation (86-2) may take Image_in2 (e.g., a processed image received from the output of the image signal processor (50-2)) as input and then warp Image_in2 using the homography (H2) to generate a corresponding second corrected and stabilized image Image_out2. When configured in this way, correction (alignment) and stabilization are applied together in a single operation to warp the image. The operations of blocks (84-2, 86-2) may be performed by the processor (60-2). Unlike the first and second processed images (Image_in1, Image_in2) received at the inputs of processors (60-1, 60-2) which may have different resolutions and different fields of view, the first and second corrected (aligned) and stabilized images (Image_out1, Image_out2) generated at the outputs of processors (60-1, 60-2) may have the same resolution and the same field of view (e.g., warped images may have the same focal length without any camera or lens distortion).

[0052] The embodiments of FIGS. 4 and 5, showing hardware and software subsystems configured to generate a stereoscopic video stream, are exemplary. FIGS. 6 and 7 illustrate other embodiments including additional circuitry for outputting stereoscopic still images (e.g., images capturing a single moment in time rather than a sequence of images). As illustrated in FIG. 6, the electronic device (10) may further include additional processing blocks such as a first denoising block (90-1), a second denoising block (90-2), and a codec block (71). The first denoising block (90-1) may have an input coupled to the output of the processor (60-1) via a data path (94-1) and may generate a first still image (Still1) by performing software-based multi-band noise reduction (as an example). The second noise removal block (90-2) may have an input coupled to the output of the processor (60-2) via the data path (94-2) and may generate a second still image (Still2) by performing software-based multi-band noise reduction (as an example). If the image resolution of Still1 and Still2 is insufficient, additional upscaling or super-resolution processing blocks may be included to increase the resolution of Still1 and Still2. The images (Still1, Still2) may be supplied to the inputs of the codec block (71). The noise removal blocks (90-1, 90-2) of FIG. 6 are optional. Generally, the blocks (90-1, 90-2) may be omitted or replaced with other image enhancement block(s) configured to perform noise removal and upscaling operations.

[0053] The codec block (71) may be configured to implement, for example, HEVC (High Efficiency Video Coding), which is a video compression protocol designed to provide efficient data compression for video content. The codec block (71) may, for example, use HEIC (High Efficiency Image Format) to generate a corresponding still stereoscopic pair (or stereoscopic image pair). This is merely illustrative. If desired, the codec block (71) may generate a compressed still stereoscopic image pair using the JPEG (Joint Photographics Expert Group) image format, PNG (Portable Network Graphics) image format, GIF (Graphics Interchange Format), TIFF (Tagged Image File Format), and / or other image formats. Thus, the codec (71) is sometimes referred to as an image compression block. If desired, the codec (71) may be configured to implement other types of encoding schemes to generate stereoscopic image pairs. A still stereoscopic pair may optionally be stored in the cloud within a remote or local database and may be played on a device having one or more displays capable of presenting stereoscopic video content. As an example, a stereoscopic video stream may be played on a head-mounted device having one or more displays for displaying slightly different content to create a sense of depth and 3D experience for the user.

[0054] The example of FIG. 6, in which a still generation circuit (e.g., blocks (90-1, 90-2), and codec (71)) utilizes corrected and stabilized images output from processors (60-1, 60-2), is exemplary. FIG. 7 illustrates another example in which a still generation circuit is directly coupled to image signal processors (50). As illustrated in FIG. 7, the device (10) may further include a first noise removal block (90-1), a second noise removal block (90-2), a third processor (60-3), and a codec block (71). The first noise removal block (90-1) may have an input coupled to an ISP (50-1) via a data path (92-1) and may perform software-based multi-band noise reduction (as an example). The second noise removal block (90-2) may have an input coupled to the output of the ISP (50-2) via the data path (92-2) and may perform software-based multi-band noise reduction (as an example). Images (e.g., still images) output on the paths (92-1, 92-2) may be generated on demand by the ISPs (50-1, 50-2). The noise removal blocks (90-1, 90-2) of FIG. 7 are optional. Generally, the blocks (90-1, 90-2) may be omitted or replaced with other image enhancement block(s) configured to provide noise removal and upscaling functions.

[0055] The processor (60-3) may have inputs coupled to the noise removal blocks (90-1, 90-2). The processors (60-3) may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a digital signal processor (DSP), a field programmable gate array (FPGA), and / or other types of processors. Unlike the processors (60-1, 60-2), which may be configured to perform both stereo correction and image stabilization functions, the processor (60-3) may be configured to perform only stereo correction functions (see subblock (64)). The processor (60-3) does not need to perform any image stabilization. The processor (60-3) can output still images (Still1, Still2) by performing stereo correction based on the output from the noise removal blocks (90-1, 90-2) (e.g., using a rotation matrix associated with the image sensor (30-1) and / or other extrinsic calibration data). Since the images output from the ISPs (50-1, 50-2) are relatively high resolution, additional upscaling or super-resolution processing blocks are not required. However, if desired, upscaling or super-resolution processing blocks may be included to increase the resolution of Still1 and Still2. The images (Still1, Still2) can be supplied to the inputs of the codec block (71).

[0056] The codec block (71) may be configured to implement, for example, HEVC (High Efficiency Video Coding), which is a video compression protocol designed to provide efficient data compression for video content. The codec block (71) may generate a corresponding still stereoscopic pair (or stereoscopic image pair) using, for example, HEIC (High Efficiency Image Format). This is merely exemplary. If desired, the codec block (71) may generate a compressed still stereoscopic image pair using the JPEG (Joint Photographics Expert Group) image format, PNG (Portable Network Graphics) image format, GIF (Graphics Interchange Format), TIFF (Tagged Image File Format), and / or other image formats. If desired, the codec (71) may be configured to implement other types of encoding schemes to generate the stereoscopic image pair. The still stereoscopic pair may optionally be stored in the cloud within a remote or local database and may be played on a device having one or more displays capable of presenting stereoscopic video content. For example, a stereoscopic video stream may be played on a head-mounted device having one or more displays for displaying slightly different content to create a sense of depth and 3D experience for the user. Various blocks illustrated in FIG. 7, such as blocks (50-1, 50-2, 60-1, 60-2, 60-3, 70, 71, 90-1, 90-2), may be collectively referred to as a control circuit (e.g., see control circuit (14) in FIG. 1).

[0057] FIG. 8 is a flowchart of exemplary steps for operating an electronic device (10) of the type illustrated in FIGS. 1 through 7 according to some embodiments. During the operations of block (100), the device (10) may be operated to start stereoscopic capture. For example, the device (10) may be operated to capture a stereoscopic video stream (e.g., using the circuitry and processes described in relation to FIGS. 4 and 5) and / or to capture a stereoscopic still image (e.g., using the circuitry and processes described in relation to FIGS. 6 and 7).

[0058] During the operations of block (102), the device (10) may optionally detect an incorrect stereoscopic capture orientation. For example, the device (10) may use one or more motion sensors or inertial measurement units to determine whether the device (10) is currently maintained in a vertical orientation or a horizontal orientation. Assuming the rear-facing cameras (30-1, 30-2) are positioned as illustrated in FIG. 3, stereoscopic capture must be performed when the device (10) is in a horizontal orientation. Thus, when the device (10) detects that the device (10) is in a vertical orientation, the device (10) may issue a notification to guide the user to use the correct capture orientation (e.g., to output a user alert to switch to a horizontal orientation, as illustrated by the operations of block (104)). If the device (10) is already in the correct (horizontal) orientation, block (104) may be skipped.

[0059] During the operations of block (106), the device (10) may optionally detect whether one of the image sensors used for stereoscopic capture is currently obscured. For example, the device (10) may determine whether one of the cameras is obscured by analyzing and comparing thumbnail information, integration time, local and / or global brightness information, color information, focus information, and / or other image statistics between images captured by two cameras. In response to determining that one of the cameras is currently obscured, the device (10) may notify the user that one of the image sensors is obscured (see operations of block (108)), and the user may be given an opportunity to remove the obscuration. For example, the user may move away from or turn away from an obstacle that may be blocking one of the cameras, move their finger that may be blocking one of the cameras, or wipe away a smudge or blemish that may be covering one of the cameras. If none of the image sensors are obscured, block (108) may be skipped. The example of FIG. 8 in which blocks (106 / 108) are performed after blocks (102 / 104) is merely illustrative. If desired, blocks (106 / 108) may be performed before or in parallel with the operations of blocks (102 / 104).

[0060] During the operations of block (110), the device (10) may optionally detect low light conditions. For example, the device (10) may determine the ambient light level of the scene being captured by analyzing brightness information collected by one of the image sensors (30-1, 30-2), by an ambient light sensor, and / or by other optical sensors. If the ambient light (lux) level of the scene being captured is below a certain threshold, the device (10) may notify the user of the low light condition (see block (112)). The user may be given the opportunity to improve the quality of the stereoscopic content being captured by adding additional lighting and / or moving to an area with better lighting conditions and / or otherwise improving the lighting conditions. If the ambient light level of the scene being captured is greater than the threshold, block (112) may be skipped. The example of FIG. 8 in which blocks (110 / 112) are performed after blocks (106 / 108) is merely exemplary. If desired, blocks (110 / 112) can be performed before or in parallel with the operations of blocks (106 / 108) or blocks (102 / 104).

[0061] During the operations of block (114), the device (10) may optionally detect a near object within the field of view of one of the image sensors (30-1, 30-2). In response to detecting a near object within the field of view of one of the rear-facing cameras used to capture stereoscopic content, the device (10) may output an alert to the user to move the object further away or remove it so that the nearby object is no longer within the field of view of one of the image sensors (30-1, 30-2) (see block (116)). If no near object is detected, block (116) may be skipped. The example of FIG. 8 in which blocks (114 / 116) are performed after blocks (110 / 112) is merely illustrative. If desired, blocks (114 / 116) may be performed before or in parallel with the operations of blocks (110 / 112, 106 / 108) or blocks (102 / 104).

[0062] During the operations of block (118), the device (10) may optionally detect whether it is moving excessively or if there is an elevated level of camera jitter. For example, the detected movement or camera jitter may be compared to a threshold level. In response to detecting excessive motion or excessive camera jitter, the device (10) may notify the user to stop moving or remain stationary (see operations of block (120)). The example of FIG. 8 in which blocks (118 / 120) are performed after blocks (114 / 116) is merely exemplary. If desired, blocks (118 / 120) may be performed before or in parallel with the operations of blocks (114 / 116, 110 / 112, 106 / 108) or blocks (102 / 104).

[0063] The operations of FIG. 8 are merely exemplary. Various blocks (102 to 120) may all serve to improve the quality of stereoscopic content. If desired, other steps to improve the quality of stereoscopic video or images may be employed. If desired, additional steps for detecting distant objects may be included. If desired, additional steps for detecting camera jitter or shaking (and associated alerts prompting the user to remain still) may be included. In some embodiments, one or more of the described operations may be modified, replaced, or omitted. In some embodiments, one or more of the described operations may be performed in parallel. In some embodiments, additional processes may be added or inserted between the described operations. If desired, the order of certain operations may be reversed or changed and / or the timing of the described operations may be adjusted so that they occur at slightly different times. In some embodiments, the described operations may be distributed across a larger system.

[0064] The methods and operations described above in relation to FIGS. 1 through 8 may be performed by the components of the device (10) using software, firmware and / or hardware (e.g., dedicated circuitry or hardware). Software code for performing these operations may be stored on a non-transient computer-readable storage medium (e.g., tangible computer-readable storage medium) stored on one or more of the components of the device (10) (e.g., storage circuitry within the control circuitry (14) of FIG. 1). Software code may sometimes be referred to as software, data, instructions, program instructions, or code. The non-transient computer-readable storage medium may include a drive, non-volatile memory such as non-volatile random access memory (NVRAM), a removable flash drive or other removable media, other types of random access memory, etc. Software stored on the non-transient computer-readable storage medium may be executed by a processing circuitry on one or more of the components of the device (10) (e.g., one or more processors within the control circuitry (14)). The processing circuit may include a microprocessor, an application processor, a digital signal processor, a central processing unit (CPU), an application integrated circuit equipped with a processing circuit, or other processing circuits.

[0065] According to one embodiment, a method for operating an electronic device is provided, comprising the steps of: capturing a first image using a first image sensor having a first field of view; capturing a second image using a second image sensor having a second field of view different from the first field of view; and outputting stereoscopic content based on the first image captured using the first image sensor having a first field of view and the second image captured using the second image sensor having a second field of view different from the first field of view.

[0066] According to another embodiment, the present method includes the step of processing a first image using a first image signal processor to output a first processed image, and the step of processing a second image using a second image signal processor to output a second processed image.

[0067] According to another embodiment, the step of processing a first image using a first image signal processor includes the step of cropping and scaling the first image, and the step of processing a second image using a second image signal processor includes the step of cropping and scaling the second image.

[0068] According to another embodiment, the method includes the step of performing stereo correction and image stabilization operations on first and second processed images to output corresponding first and second corrected and stabilized images.

[0069] According to another embodiment, the present method includes the step of compressing first and second corrected and stabilized images using a multi-view video encoding method to generate a stereoscopic video stream.

[0070] According to another embodiment, the step of performing stereo correction and image stabilization operations on the first and second processed images includes the step of obtaining stabilization information for the first image sensor, and the step of synchronizing image stabilization between the first and second image sensors by applying the stabilization information for the first image sensor to the second processed image captured using the second image sensor.

[0071] According to another embodiment, the step of performing stereo correction and image stabilization operations on the first and second processed images includes the step of acquiring a rotation matrix associated with the first image sensor, and the step of calculating a corrected and stabilized pose based on the rotation matrix associated with the first image sensor and motion data.

[0072] According to another embodiment, the step of calculating a corrected and stabilized pose includes the step of correcting the pose of a first image sensor and the step of stabilizing the corrected pose of the first image sensor using a time filter.

[0073] According to another embodiment, the present method comprises the step of calculating a first homography based on a corrected and stabilized pose, motion data associated with a first image sensor, intrinsic data associated with a first image sensor, and intrinsic data associated with a target image sensor.

[0074] According to another embodiment, the method comprises the step of calculating a second homography based on a corrected and stabilized pose, motion data associated with a second image sensor, intrinsic data associated with a second image sensor, and intrinsic data associated with a target image sensor.

[0075] According to another embodiment, the present method comprises the steps of: warping a first processed image using a first homography to generate a first corrected and stabilized image; and warping a second processed image using a second homography to generate a second corrected and stabilized image.

[0076] According to another embodiment, the present method comprises the steps of: outputting a first still image by denoising and upscaling a first corrected and stabilized image; outputting a second still image by denoising and upscaling a second corrected and stabilized image; and generating a still stereoscopic pair by compressing the first and second still images.

[0077] According to another embodiment, the method comprises the steps of denoising and upscaling a first additional image obtained from a first image signal processor, denoising and upscaling a second additional image obtained from a second image signal processor, performing stereo correction on the first and second additional images to output first and second still images, and generating a still stereoscopic pair by compressing the first and second still images.

[0078] According to one embodiment, an electronic device is provided comprising: a first camera configured to capture a first image having a first field of view; a second camera configured to capture a second image having a second field of view different from the first field of view; and a control circuit configured to output stereoscopic content based on a first image captured from a first camera having a first field of view and a second image from a second camera having a second field of view different from the first field of view.

[0079] According to another embodiment, the control circuit includes a first image signal processor configured to receive a first image captured by a first camera and output a corresponding first processed image, and a second image signal processor configured to receive a second image captured by a second camera and output a corresponding second processed image.

[0080] According to another embodiment, the control circuit comprises a first additional processor configured to receive a first processed image, motion data associated with a first camera, and first calibration data, and further configured to generate a first calibrated and stabilized image, and a second additional processor configured to receive a second processed image, motion data associated with a second camera, and second calibration data, and further configured to generate a second calibrated and stabilized image.

[0081] According to another embodiment, the control circuit includes a video compression block configured to receive first and second corrected and stabilized images and generate a corresponding stereoscopic video stream.

[0082] According to another embodiment, the control circuit comprises a first image enhancement block configured to remove noise and upscale a first corrected and stabilized image, a second image enhancement block configured to remove noise and upscale a second corrected and stabilized image, and an image compression block coupled to the outputs of the first and second noise removal blocks.

[0083] According to another embodiment, the control circuit comprises a first image enhancement block configured to receive a first additional image from a first image signal processing block and perform noise removal and upscaling operations; a second image enhancement block configured to receive a second additional image from a second image signal processing block and perform noise removal and upscaling operations; a third additional processor configured to be coupled to the outputs of the first and second noise removal blocks and to perform stereo correction operations to output corresponding first and second still images; and an image compression block configured to receive the first and second still images from the third additional processor.

[0084] According to one embodiment, a method for operating an electronic device is provided, comprising the steps of: capturing a first image using a first camera having a first field of view; capturing a second image using a second camera having a second field of view different from the first field of view; generating stereoscopic content based on the first and second captured images; and outputting a user notification to improve the quality of the stereoscopic content.

[0085] According to another embodiment, the step of outputting a user notification to improve the quality of stereoscopic content includes detecting an incorrect stereoscopic capture orientation of an electronic device, and outputting a user notification to switch to a correct stereoscopic capture orientation in response to detecting the incorrect stereoscopic capture orientation of the electronic device.

[0086] According to another embodiment, the step of outputting a user notification for improving the quality of stereoscopic content includes detecting whether the first camera is obscured, and in response to detecting that the first camera is obscured, outputting a user notification that the first camera is obscured.

[0087] According to another embodiment, the step of outputting a user notification for improving the quality of stereoscopic content includes detecting the lighting conditions of a first or second image, and outputting a user notification that the lighting conditions are below a threshold in response to detecting that the detected lighting is below a threshold.

[0088] According to another embodiment, the step of outputting a user notification to improve the quality of stereoscopic content includes the step of detecting an external object within the field of view of the first camera, and the step of outputting a user notification that the external object is within the field of view of the first camera in response to detecting the external object within the field of view of the first camera.

[0089] According to another embodiment, the step of outputting a user notification to improve the quality of stereoscopic content includes detecting motion or jitter of the first and second cameras, and outputting a user notification to stop moving or remain stationary in response to detecting motion or jitter of the first and second cameras.

[0090] The foregoing is merely illustrative, and various modifications may be made to the described embodiments. The foregoing embodiments may be implemented individually or in any combination.

Claims

Claim 1 A method for operating an electronic device, comprising: capturing a first image using a first image sensor having a first field of view; capturing a second image using a second image sensor having a second field of view different from the first field of view; processing the first image using a first image signal processor to output a first processed image; processing the second image using a second image signal processor to output a second processed image; obtaining a rotation matrix associated with the first image sensor and calculating a corrected and stabilized pose based on the rotation matrix and motion data associated with the first image sensor, thereby performing stereo rectification and image stabilization operations on the first and second processed images to output corresponding first and second corrected and stabilized images; and outputting stereoscopic content based on the first and second corrected and stabilized images. Claim 2 delete Claim 3 A method according to claim 1, wherein the step of processing the first image using the first image signal processor includes the step of cropping and scaling the first image, and the step of processing the second image using the second image signal processor includes the step of cropping and scaling the second image. Claim 4 delete Claim 5 A method for operating an electronic device, comprising: capturing a first image using a first image sensor having a first field of view; capturing a second image using a second image sensor having a second field of view different from the first field of view; processing the first image using a first image signal processor to output a first processed image; processing the second image using a second image signal processor to output a second processed image; performing stereo rectification and image stabilization operations on the first and second processed images to output corresponding first and second rectified and stabilized images; compressing the first and second rectified and stabilized images using a multi-view video encoding method to generate a stereoscopic video stream; and outputting stereoscopic content based on the first and second rectified and stabilized images. Claim 6 A method for operating an electronic device, comprising: a step of capturing a first image using a first image sensor having a first field of view; a step of capturing a second image using a second image sensor having a second field of view different from the first field of view; a step of processing the first image using a first image signal processor to output a first processed image; a step of processing the second image using a second image signal processor to output a second processed image; and a step of performing stereo rectification and image stabilization operations on the first and second processed images to output corresponding first and second rectified and stabilized images — the step of performing the stereo rectification and image stabilization operations on the first and second processed images is achieved by obtaining stabilization information for the first image sensor and synchronizing image stabilization between the first and second image sensors by applying the stabilization information for the first image sensor to the second processed image captured using the second image sensor —; A method comprising the step of outputting stereoscopic content based on the first and second corrected and stabilized images. Claim 7 delete Claim 8 A method according to claim 1, wherein the step of calculating the corrected and stabilized pose comprises: a step of correcting the pose of the first image sensor; and a step of stabilizing the corrected pose of the first image sensor using a time filter. Claim 9 A method according to claim 1, further comprising the step of calculating a first homography based on the corrected and stabilized pose, the motion data associated with the first image sensor, the intrinsic data associated with the first image sensor, and the intrinsic data associated with the target image sensor. Claim 10 The method of claim 9 further comprises the step of calculating a second homography based on the corrected and stabilized pose, motion data associated with the second image sensor, intrinsic data associated with the second image sensor, and the intrinsic data associated with the target image sensor. Claim 11 A method according to claim 10, further comprising: a step of warping the first processed image using the first homography to generate the first corrected and stabilized image; and a step of warping the second processed image using the second homography to generate the second corrected and stabilized image. Claim 12 A method according to claim 11, further comprising the step of denoising and upscaling the first corrected and stabilized image to output a first still image, denoising and upscaling the second corrected and stabilized image to output a second still image, and compressing the first and second still images to generate a still stereoscopic pair. Claim 13 A method according to claim 11, further comprising the step of denoising and upscaling a first additional image obtained from the first image signal processor, denoising and upscaling a second additional image obtained from the second image signal processor, performing stereo correction on the first and second additional images to output first and second still images, and generating a still stereoscopic pair by compressing the first and second still images. Claim 14 An electronic device comprising: a first camera configured to capture a first image having a first field of view; a second camera configured to capture a second image having a second field of view different from the first field of view; and a control circuit configured to output stereoscopic content based on the first image captured from the first camera having the first field of view and the second image from the second camera having the second field of view different from the first field of view, generate a first still image by denoising and upscaling a first processed image generated based on the first image, generate a second still image by denoising and upscaling a second processed image generated based on the second image, and compress the first and second still images. Claim 15 An electronic device according to claim 14, wherein the control circuit comprises: a first image signal processor configured to receive the first image captured by the first camera and output the first processed image; and a second image signal processor configured to receive the second image captured by the second camera and output the second processed image. Claim 16 An electronic device according to claim 15, wherein the control circuit comprises: a first additional processor configured to receive the first processed image, motion data associated with the first camera, and first calibration data, and further configured to generate a first corrected and stabilized image; and a second additional processor configured to receive the second processed image, motion data associated with the second camera, and second calibration data, and further configured to generate a second corrected and stabilized image. Claim 17 An electronic device according to claim 16, wherein the control circuit further comprises a video compression block configured to receive the first and second corrected and stabilized images and generate a corresponding stereoscopic video stream. Claim 18 An electronic device comprising: a first camera having a first field of view and configured to capture a first image; a second camera having a second field of view different from the first field of view and configured to capture a second image; and a control circuit configured to output stereoscopic content based on the first image captured from the first camera having the first field of view and the second image from the second camera having the second field of view different from the first field of view, wherein the control circuit comprises: a first image signal processor configured to receive the first image captured by the first camera and output a first processed image; a second image signal processor configured to receive the second image captured by the second camera and output a second processed image; and a first additional processor configured to receive the first processed image, motion data associated with the first camera, and first calibration data, and further configured to generate a first corrected and stabilized image. An electronic device comprising: a second additional processor configured to receive the second processed image, motion data associated with the second camera, and second calibration data, and further configured to generate a second corrected and stabilized image; a first image enhancement block configured to remove noise and upscale the first corrected and stabilized image; a second image enhancement block configured to remove noise and upscale the second corrected and stabilized image; and an image compression block coupled to the outputs of the first and second noise removal blocks. Claim 19 An electronic device comprising: a first camera having a first field of view and configured to capture a first image; a second camera having a second field of view different from the first field of view and configured to capture a second image; and a control circuit configured to output stereoscopic content based on the first image captured from the first camera having the first field of view and the second image from the second camera having the second field of view different from the first field of view, wherein the control circuit comprises: a first image signal processor configured to receive the first image captured by the first camera and output a first processed image; a second image signal processor configured to receive the second image captured by the second camera and output a second processed image; and a first additional processor configured to receive the first processed image, motion data associated with the first camera, and first calibration data, and further configured to generate a first corrected and stabilized image. An electronic device comprising: a second additional processor configured to receive the second processed image, motion data associated with the second camera, and second calibration data, and further configured to generate a second corrected and stabilized image; a first image enhancement block configured to receive a first additional image from a first image signal processing block and perform noise removal and upscaling operations; a second image enhancement block configured to receive a second additional image from a second image signal processing block and perform noise removal and upscaling operations; a third additional processor coupled to the outputs of the first and second noise removal blocks and configured to perform stereo correction operations to output corresponding first and second still images; and an image compression block configured to receive the first and second still images from the third additional processor. Claim 20 A method for operating an electronic device, comprising: capturing a first image using a first camera having a first field of view; capturing a second image using a second camera having a second field of view different from the first field of view; determining whether the electronic device is in a first capture orientation; outputting a user notification to switch to a second capture orientation different from the first capture orientation in response to the determination that the electronic device is in the first capture orientation; and generating stereoscopic content based on the first and second captured images after the electronic device has switched from the first capture orientation to the second capture orientation. Claim 21 delete Claim 22 A method according to claim 20, further comprising: a step of detecting whether the first camera is obscured; and a step of outputting a user notification that the first camera is obscured in response to detecting that the first camera is obscured. Claim 23 A method according to claim 20, further comprising: a step of detecting a lighting condition of the first or second image; and a step of outputting a user notification that the lighting condition is less than the threshold in response to detecting that the detected lighting is less than the threshold. Claim 24 A method according to claim 20, further comprising: a step of detecting an external object within the field of view of the first camera; and a step of outputting a user notification that the external object is within the field of view of the first camera in response to detecting the external object within the field of view of the first camera. Claim 25 A method according to claim 20, further comprising: a step of detecting motion or jitter of the first and second cameras; and a step of outputting a user notification to stop moving or remain stationary in response to detecting motion or jitter of the first and second cameras. Claim 26 A method according to claim 20, wherein the first capture orientation is the portrait orientation of the electronic device and the second capture orientation is the landscape orientation of the electronic device. Claim 27 A method for operating an electronic device, comprising: capturing a first image using a first image sensor having a first field of view; capturing a second image using a second image sensor having a second field of view different from the first field of view; cropping and scaling the first image using a first image signal processor to generate a first processed image having a first resolution; cropping and scaling the second image using a second image signal processor to generate a second processed image having a second resolution different from the first resolution; warping the first processed image to generate a first corrected and stabilized image having a third resolution and a third field of view; warping the second processed image to generate a second corrected and stabilized image having the third resolution and the third field of view; and outputting stereoscopic content based on the first corrected and stabilized image and the second corrected and stabilized image. Claim 28 A method according to claim 27, further comprising the step of compressing the first and second corrected and stabilized images using a multi-view video encoding method to generate a stereoscopic video stream for display. Claim 29 A method according to claim 27, further comprising: a step of obtaining stabilization information for the first image sensor; and a step of synchronizing image stabilization between the first and second image sensors by applying the stabilization information for the first image sensor to the second processed image captured using the second image sensor. Claim 30 A method according to claim 27, further comprising the steps of: obtaining a rotation matrix associated with the first image sensor; and calculating a corrected and stabilized pose of the first image sensor based on the rotation matrix and motion data associated with the first image sensor. Claim 31 An electronic device comprising: a first image sensor configured to capture a first image having a first field of view; a second image sensor configured to capture a second image having a second field of view wider than the first field of view; a first image signal processor configured to crop and scale the first image to generate a first processed image having a first resolution; a second image signal processor configured to crop and scale the second image to generate a second processed image having a second resolution different from the first resolution; a first processing circuit configured to warp the first processed image to generate a first corrected and stabilized image having a third resolution and a third field of view; a second processing circuit configured to warp the second processed image to generate a second corrected and stabilized image having the third resolution and the third field of view; and a compression circuit configured to output stereoscopic content based on the first corrected and stabilized image and the second corrected and stabilized image.

Images