Foldable electronic device for multi-view image capture

Abstract

Methods, systems, and devices are provided that automatically adjust a position of an image sensor and generate a multi-view image based on image data captured from the image sensor. For example, an image capture device includes a housing that includes a first housing portion and a second housing portion, where a first sensor is coupled to the first housing portion and a second sensor is coupled to the second housing portion. A coupling device, such as a hinge, couples the first housing portion to the second housing portion. The image capture device obtains image data from the first sensor and the second sensor and determines an object depth based on the obtained image data. The image capture device outputs an adjusted angle based on the object depth and obtains additional image data from the first sensor and the second sensor. The image capture device generates a multi-view image based on the additional image data.

Description

Technical Field

[0001] This disclosure relates generally to imaging devices, and more specifically to foldable display devices with image capture capabilities. Background Technology

[0002] Image capture devices such as phones, tablets, and smart devices may include an outward-facing camera for capturing images. For example, the camera may be located on an outward-facing surface opposite to an inward-facing surface, where the inward-facing surface includes a display. The image captured from the camera on the outward-facing surface is displayed on the display. Image capture devices may also include a camera on an inward-facing surface to facilitate capturing "selfies." Some of these image capture devices include foldable displays. Each side of a foldable display includes an outward-facing surface and an inward-facing surface. Image capture devices with foldable displays may include one or more cameras on each outward-facing surface.

[0003] In addition, image capture devices employ various signal processing techniques to attempt to render high-quality images. For example, an image capture device can automatically focus its lens to achieve image sharpness, automatically set exposure time based on light levels, and automatically adjust white balance to adapt to the color temperature of the light source. In some examples, the image capture device can generate multi-view images based on images captured from multiple cameras. In some examples, the image capture device includes face detection technology. Face detection technology allows the image capture device to identify faces within the field of view of the camera lens. The image capture device can then apply various signal processing techniques based on the identified faces. Summary of the Invention

[0004] According to one aspect, a method for operating an image capture device includes: acquiring first image data from a first sensor coupled to a first portion of a housing of the image capture device. The method further includes: acquiring second image data from a second sensor coupled to a second portion of the housing. The method further includes: determining the depth of an object based on the first image data and the second image data. The method further includes: outputting an adjusted angle between the first housing portion and the second housing portion based on the determined depth. Furthermore, the method includes: performing an image capture operation in response to the adjustment.

[0005] According to another aspect, an image capture device includes a housing comprising a first housing portion and a second housing portion. The image capture device further includes a non-transitory machine-readable storage medium storing instructions, and at least one processor coupled to the non-transitory machine-readable storage medium. The at least one processor is configured to execute the instructions to acquire first image data from a first sensor coupled to the first housing portion of the housing. The processor is also configured to execute the instructions to acquire second image data from a second sensor coupled to the second housing portion of the housing. Furthermore, the processor is configured to execute the instructions to determine the depth of an object based on the first image data and the second image data. The processor is also configured to execute instructions to output an adjusted angle between the first housing portion and the second housing portion based on the depth. The processor is further configured to execute the instructions to perform an image capture operation based on the adjusted angle.

[0006] According to another aspect, a non-transitory machine-readable storage medium stores instructions that, when executed by at least one processor, cause the at least one processor to perform operations, the operations including: acquiring first image data from a first sensor coupled to a first housing portion of a housing of an image capture device; acquiring second image data from a second sensor coupled to a second housing portion of the housing; determining a depth of an object based on the first image data and the second image data; outputting an adjusted angle between the first portion and the second portion based on the depth; and performing an image capture operation based on the adjusted angle.

[0007] According to another aspect, an image capture device includes: a unit for obtaining first image data from a first sensor coupled to a first housing portion of a housing of the image capture device; a unit for obtaining second image data from a second sensor coupled to a second housing portion of the housing; a unit for determining the depth of an object based on the first image data and the second image data; a unit for outputting an adjusted angle between the first portion and the second portion based on the depth; and a unit for performing an image capture operation based on the adjusted angle. Attached Figure Description

[0008] Figure 1 This is a block diagram of an exemplary image capture device according to some implementation methods;

[0009] Figure 2 This illustrates according to some embodiments. Figure 1 A block diagram of exemplary components of an image capture device;

[0010] Figure 3A , 3B3C and 3D are schematic diagrams illustrating components of an exemplary image capture device according to some embodiments;

[0011] Figure 4A , 4B 4C and 4D are schematic diagrams illustrating an exemplary image capture device with a foldable display according to some embodiments;

[0012] Figure 5 and 6 This is a flowchart illustrating an exemplary process for performing an image capture operation in an image capture device according to some embodiments; and

[0013] Figure 7 This is a flowchart of an exemplary process for generating a user-synthesized image in an image capture device, according to some implementation methods. Detailed Implementation

[0014] While the features, methods, apparatuses, and systems described herein may be embodied in various forms, some exemplary and non-limiting embodiments are shown in the accompanying drawings and described below. Some components described in this disclosure are optional, and some implementations may include additional, different, or fewer components compared to those explicitly described in this disclosure.

[0015] In some embodiments, the image capture device may be foldable and may include a foldable display along an inner surface. For example, the image capture device may be foldable along a central portion including one or more connecting means (e.g., hinges), and the central portion may separate a first portion (e.g., the left side) of the image capture device from a second portion (e.g., the right side). Each of the first and second portions may include an inner surface and an outer surface (e.g., a first inner surface and a second inner surface). For example, when the image capture device is closed (e.g., shut), the respective inner surfaces may face each other as the first and second portions of the image capture device face each other. When closed, the outer surfaces of the first and second portions face outwards, and the inner surfaces of the first and second portions may establish or define the boundaries of a cavity or void space (e.g., they may form a cavity). The foldable display may also include a corresponding portion along each inner surface of the first and second portions of the image capture device. The inner surfaces (and the foldable display) may be presented on the "user-facing" side of the image capture device.

[0016] The image capturing device may include one or more cameras on the inner surface of the first portion and one or more cameras on the inner surface of the second portion. In some examples, the image capturing device may include one or more cameras on the outer surface of the first portion. In some examples, the image capturing device may include one or more cameras on the outer surface of the second portion.

[0017] The image capture device can capture image data from each camera. For example, the image capture device can capture first image data from a first camera located on the inner surface of a first portion (i.e., the first inner surface), and can also capture second image data from a second camera located on the inner surface of a second portion (i.e., the second inner surface). The first and second cameras can together form a stereo camera (e.g., a left camera and a right camera).

[0018] The image capture device can determine the depth of an object based on first image data and second image data. For example, the image capture device can identify objects in each field of view (FOV) of the first and second cameras, and can selectively provide a lens position that includes a focus value of the object's region of interest (ROI). Based on the determined depth, the image capture device can automatically adjust one or more coupling devices to change the angle separating the first and second parts (i.e., the angle of the one or more coupling devices). In response to this adjustment, the angle can increase (e.g., when the image capture device is open), or the angle can decrease (e.g., when the image capture device is closed).

[0019] In some examples, the image capture device provides visual or auditory instructions to the user of the image capture device to adjust the angle (e.g., by further closing or further opening the image capture device).

[0020] In some examples, the image capture device determines the current angle and adjusts the angle based on the depth value and the current angle.

[0021] In some examples, the image capture device determines the depth of an object based on camera parameters. For each camera, camera parameters can include intrinsic and extrinsic parameters. The intrinsic and extrinsic parameters can be predetermined and stored in a data repository maintained by the image capture device (e.g., in tangible, non-transitory memory).

[0022] Intrinsic parameters can characterize the transformation from image plane coordinates to pixel coordinates for each of the first and second cameras. For example, for each of the first and second cameras, intrinsic parameters may include focal length, image sensor format, and principal point. Extrinsic parameters characterize the relative position and orientation of each of the first or second cameras. For example, extrinsic parameters may include rotation and translation matrices that characterize the relative position and orientation of one or more cameras in the first and second cameras, respectively. For example, a translation matrix may identify a translation from one camera's reference frame to another camera's reference frame. For example, a first translation matrix may identify a translation of the reference frame from the first camera to the second camera, and a second translation matrix may identify a translation of the reference frame from the second camera to the first camera. Rotation matrices can identify rotations from one camera's reference frame to another camera's reference frame. For example, a first rotation matrix may identify a rotation of the reference frame from the first camera to the second camera, and a second rotation matrix may identify a rotation of the reference frame from the second camera to the first camera.

[0023] In addition, external parameters may include the baseline lengths of the first and second cameras. The baseline length defines the distance between the optical centers of the first and second cameras. In some examples, the larger the angle between the first and second inner surfaces, the larger the baseline length between the first and second cameras. As an example, a first angle between the first and second inner surfaces of the image capture device can establish a first baseline length between the first and second cameras, and a second angle between the first and second inner surfaces of the image capture device can establish a second baseline length between the first and second cameras. For example, if the first angle exceeds the second angle, the established first baseline length can also exceed and be greater than the corresponding second baseline length.

[0024] An image capture device can align first image data from a first camera with second image data from a second camera based on a rotation matrix, a translation matrix, and, alternatively, a baseline length. For example, for a given angle between the first and second inner surfaces, external parameters can specify the rotation and translation matrices for each of the first and second cameras, as well as the corresponding baseline length. The image capture device can determine the current angle and can align the first image data from the first camera with the second image data from the second camera based on the rotation matrix, the translation matrix, and, alternatively, the baseline length corresponding to the current angle. In some examples, the image capture device can determine the external parameters during calibration of each of the first and second cameras prior to use.

[0025] The image capture device can also identify features (e.g., objects) in each of the first and second image data. According to some examples, pixels in the first image data that have features (e.g., edges, brightness, color) are matched with pixels in the second image data that have corresponding features. Based on the matched pixels in the first and second image data, the image capture device can generate a disparity map (e.g., a stereo depth map) using any technique known in the art. For example, the disparity map can encode the difference in the horizontal coordinates of corresponding features and can include disparity values, where each disparity value is inversely proportional to the scene depth at the corresponding pixel location.

[0026] The image capture device can calculate depth values ​​based on the intrinsic parameters of each of the first and second cameras. For example, the image capture device can determine depth values ​​based on the focal length, image sensor format, and principal point of each of the first and second cameras. In some examples, the image capture device can calculate depth values ​​based on rotation matrices used for the first and second cameras. In some examples, the image capture device calculates depth values ​​based on features classified within first image data acquired from the first camera and second image data acquired from the second camera, and based on portions of the disparity map. The image capture device can also calculate depth values ​​based on the focal length of the first camera, the focal length of the second camera, a combination of the focal lengths of the first and second cameras, corresponding baseline lengths, portions of the disparity map, and so on.

[0027] In some examples, the computing device includes a housing having a first portion and a second portion, wherein the first portion forms an angle relative to the second portion. Furthermore, the first portion of the housing may include a first sensor, and the second portion of the housing may include a second sensor. The computing device can determine a first angle between the first and second portions of the housing, and also determine a baseline length between the first and second sensors based on the first angle. Additionally, the computing device can determine the focal length of at least one of the first and second sensors. The computing device can generate a depth value based on the baseline length, focal length, and a portion of the disparity map.

[0028] Furthermore, the image capture device can adjust the angle between the first inner surface and the second inner surface based on the calculated depth value. For example, the image capture device can identify an object based on classified features and calculate the object's depth value as described above. The image capture device can calculate the object depth based on the depth value. Furthermore, the image capture device can determine the desired baseline length between the first and second cameras for capturing an image of the identified object based on the object depth. For example, the image capture device can calculate a larger baseline length for objects with a larger object depth and a smaller baseline length for objects with a smaller object depth. In some examples, the image capture device maintains one or more tables in a memory device that identify matches between the desired baseline length and the object depth of a specific object. For example, a first table might identify the desired baseline length for each of multiple object depths of an object classified as a "face," and a second table might identify the desired baseline length for each of multiple object depths of an object classified as a "tree."

[0029] Based on the desired baseline length, the image capture device can determine the adjustment of the angle between the first inner surface and the second inner surface. For example, the image capture device can determine the current baseline length between the first camera and the second camera, and can compare the current baseline length with the desired baseline length to determine the angle adjustment amount. The image capture device can then adjust the angle based on the determined angle adjustment amount.

[0030] In some examples, after adjusting the angle, the image capture device captures third image data from a first camera and fourth image data from a second camera, and generates a multi-view image based on the third and fourth image data. For example, the image capture device can identify features in the third and fourth image data and generate a second disparity map based on the identified features. The image capture device can use any known technique to generate the multi-view image based on portions of the third image data, the fourth image data, and the second disparity map. The multi-view image may, for example, include a three-dimensional (3D) image. In other examples, the multi-view image may include a "high dynamic range" image or a "non-high dynamic range" image.

[0031] In some examples, the image capture device can periodically determine whether to adjust the angle while the user is capturing an image of the scene. For example, the image capture device can periodically acquire image data from each of a first camera and a second camera, and adjust the angle as described herein, until a predetermined amount of time before capturing image data to generate a multi-view image.

[0032] In some implementations, the image capture device can determine the Region of Interest (ROI) after adjusting the angle for improved autofocus (AF), auto exposure (AE), auto gain (AG), or auto white balance (AWB) control. For example, the image capture device can adjust the angle as described herein, identify features of an object (e.g., within third image data and additional image data), and determine an ROI that includes the object's face. The image capture device can then determine (e.g., adjust, apply) one or more of AF, AE, AG, or AWB controls based on the image data within the ROI.

[0033] In some examples, the image capture device can provide automatic image capture enhancement (e.g., 3D image enhancement) to the multi-view images generated in response to angle adjustments. For example, the image capture device can apply lighting effects to the multi-view images to generate user-synthesized images. For example, the user can select the lighting effects used for image capture. Examples of lighting effects include color shift, segmentation effects (e.g., depth of field effect, bokeh effect, blue screen, object filters such as face filters, etc.), augmented reality (AR), virtual reality (VR), and averaging effects. Averaging effects can, for example, average the color channel values ​​of all pixels or selected groups of pixels in two different images. Depth of field effects (e.g., bokeh effect) can depend on the color channel values ​​of pixels in a third image data and pixels in a fourth image data.

[0034] Figure 1 This is a block diagram of an exemplary image capture device 100. The functionality of the image capture device 100 may be implemented in one or more processors, one or more field-programmable gate arrays (FPGAs), one or more application-specific integrated circuits (ASICs), one or more state machines, digital circuits, any other suitable circuitry, or any suitable hardware.

[0035] In this example, the image capture device 100 includes at least one processor 160 operatively coupled (e.g., in communication with) camera optics and sensors 115A and 115B, respectively, for capturing corresponding images. Camera optics and sensors 115A and 115B may each include one or more image sensors and one or more lenses for image capture. Camera optics and sensors 115A may be disposed within a first portion 117A of the image capture device 100, and camera optics and sensors 115B may be disposed within a second portion 117B of the image capture device 100. Each of the camera optics and sensors 115A and 115B may be based on, for example, a time-of-flight (TOF) or structured light (e.g., infrared, random point projection) image capture system. The processor 160 is also operatively coupled to a hinge control 119, which allows control (e.g., mechanical adjustment) of the hinge angle of the hinge connecting the first portion 117A to the second portion 117B. For example, processor 160 can control hinge control 119 to adjust the hinge angle between 0 degrees and 180 degrees (inclusive).

[0036] The processor 160 is also operatively coupled to the instruction memory 130, the working memory 105, the input device 170, the transceiver 111, the foldable display 125, and the storage medium 110. The input device 170 may be, for example, a keyboard, a touchpad, a stylus, a touch screen, or any other suitable input device.

[0037] Image capture device 100 may be implemented in a computer with image capture capability, a dedicated camera, a multi-purpose device capable of performing imaging and non-imaging applications, or any other suitable device. For example, image capture device 100 may include a portable personal computing device, such as a mobile phone (e.g., a smartphone), a tablet computer, a personal digital assistant, or any other suitable device.

[0038] Processor 160 may also include one or more processors. For example, processor 160 may include one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more digital signal processors (DSPs), one or more image signal processors (ISPs), one or more device processors, and / or one or more other suitable processors. Processor 160 may also perform various image capture operations on the received image data to perform AF, AG, AE, and / or AWB. In addition, processor 160 may perform various management tasks, such as controlling the foldable display 125 to display the captured image, or writing or reading data to or from working memory 105 or storage medium 110. For example, processor 160 may obtain internal parameters 167 and external parameters 169 for each camera from storage medium 110. Storage medium 110 may also store image capture parameters for image capture operations, such as AF, AE, and / or AWB parameters. Processor 160 may apply AF, AE, and / or AWB based on the image capture parameters.

[0039] The processor 160 can store the captured images in the storage medium 110. For example, the processor 160 can obtain image sensor data 165 from camera optics and sensors 115A and 115B, and can store the obtained image sensor data 165 in the storage medium 110.

[0040] In some examples, processor 160 can apply light effects to the captured image. For example, processor 160 can apply color shift, depth of field (e.g., bokeh effect), augmented reality (AR), virtual reality (VR), and averaging effects to the captured image.

[0041] In some cases, transceiver 111 may use any suitable communication protocol to facilitate cross-communication network communication between image capture device 100 and one or more network-connected computing systems or devices. Examples of such communication protocols include, but are not limited to, cellular communication protocols (such as code division multiple access). Global System for Mobile Communications or Wideband Code Division Multiple Access ) and / or wireless LAN protocols (such as IEEE 802.11) or global microwave access interoperability ).

[0042] Processor 160 can control each of the camera optics and sensors 115A and 115B to capture images. For example, processor 160 can instruct camera optics and sensor 115A to initiate image capture and capture corresponding image data (e.g., take a picture), and processor 160 can receive the captured image data from camera optics and sensor 115A. Similarly, processor 160 can instruct camera optics and sensor 115B to initiate the capture of additional images and capture corresponding image data, and processor 160 can receive the captured image data from camera optics and sensor 115B.

[0043] In some examples, camera optics and sensor 115A, storage medium 110, and processor 160 provide units for capturing first image data from a first front-facing camera disposed within a first internal portion (e.g., first portion 117A) of the foldable display (e.g., foldable display 125). In some camera optics and sensor 115B, storage medium 110 and processor 160 provide units for capturing second image data from a second front-facing camera disposed within a second internal portion (e.g., second portion 117B) of the foldable display (e.g., foldable display 125).

[0044] Instruction memory 130 may store instructions that can be accessed (e.g., read) and executed by processor 160. For example, instruction memory 130 may include read-only memory (ROM), such as electrically erasable programmable read-only memory (EEPROM), flash memory, removable disk, CD-ROM, any non-volatile memory, or any other suitable memory.

[0045] Processor 160 can store data in and read data from working memory 105. For example, processor 160 can store a working set of instructions in working memory 105, such as instructions loaded from instruction memory 130. Processor 160 can also use working memory 105 to store dynamic data generated during the operation of image capture device 100. Working memory 105 can be random access memory (RAM), such as static random access memory (SRAM) or dynamic random access memory (DRAM), or any other suitable memory.

[0046] In this example, instruction memory 130 stores capture control instructions 135, autofocus (AF) instructions 140, automatic white balance (AWB) instructions 141, automatic exposure (AE) instructions 142, automatic gain (AG) instructions 148, image processing instructions 143, a face detection engine 144, a depth determination engine 146, a hinge angle control engine 147, a brightness detection engine 149, a brightness-based dynamic range detection engine 151, a lighting effects engine 153, and operating system instructions 145. Instruction memory 130 may also include additional instructions for configuring processor 160 to perform various image processing and device management tasks.

[0047] AF instruction 140 may include instructions, when executed by processor 160, to adjust the position of the camera optics and the lens of sensor 115A or 115B to the corresponding lens position. For example, processor 160 may adjust the camera optics and the lens of sensor 115A so that light from the ROI within the FOV of the imaging sensor is focused in the plane of the sensor. The selected ROI may correspond to one or more focal points of the AF system. AF instruction 140 may include instructions, when executed by processor 160, to perform autofocus operations, such as finding the optimal lens position for focusing light from the ROI in the plane of the sensor. Autofocus may include, for example, phase detection autofocus (PDAF), contrast autofocus, or laser autofocus.

[0048] AWB instruction 141 may include instructions that, when executed by processor 160, cause processor 160 to determine color correction to be applied to an image. For example, the executed AWB instruction 141 may cause processor 160 to determine the average color temperature of the illumination source under which the image is captured by camera optics and sensor 115A or camera optics and sensor 115B, and to scale the color components (e.g., R, G, and B) of the captured image so that they conform to the light under which the image is to be displayed or printed. Furthermore, in some examples, the executed AWB instruction 141 may cause processor 160 to determine the illumination source in a region of interest (ROI) of the image. Processor 160 may then apply color correction to the image based on the determined color temperature of the illumination source in the ROI of the image.

[0049] The AG instruction 148 may include instructions that, when executed by the processor 160, cause the processor 160 to determine the gain correction to be applied to the image. For example, the executed AG instruction 148 may cause the processor 160 to amplify the signal received from the lens of the camera optics and sensor 115A or the camera optics and sensor 115B. The executed AG instruction 148 may also cause the processor 160 to adjust pixel values ​​(e.g., digital gain).

[0050] AE instruction 142 may include instructions, when executed by processor 160, to determine the length of time that one or more sensing elements (such as the imaging sensors of camera optics and sensors 115A or 115B) integrate light before capturing an image. For example, the executed AE instruction 142 may cause processor 160 to meter ambient light and select an exposure time for the lens based on the metering of ambient light. The selected exposure time becomes shorter as the ambient light level increases and longer as the ambient light level decreases. For example, in the case of a digital single-lens reflex (DSLR) camera, the executed AE instruction 142 may cause processor 160 to determine the exposure speed. In another example, the executed AE instruction 142 may cause processor 160 to meter the ambient light in the ROI of the field of view of the sensors of camera optics and sensors 115A or 115B.

[0051] The capture control command 135 may include instructions that, when executed by the processor 160, cause the processor 160 to adjust the lens position, set the exposure time, set the sensor gain, and / or configure the white balance filter of the image capture device 100. The capture control command 135 may also include instructions that, when executed by the processor 160, control the overall image capture function of the image capture device 100. For example, the executed capture control command 135 may cause the processor 160 to execute an AF command 140, which causes the processor 160 to calculate lens or sensor movement to achieve a desired autofocus position and output lens control signals to control the lens of camera optics and sensor 115A or camera optics and sensor 115B.

[0052] Operating system 145 may include instructions that cause processor 160 to implement the operating system when executed by processor 160. The operating system may act as an intermediary between programs (such as user applications) and processor 160. Operating system instructions 145 may include device drivers for managing hardware resources (such as camera optics and sensors 115A or 115B, foldable display 125, or transceiver 111). Furthermore, as discussed herein, one or more of the executed image processing instructions 143 may interact indirectly with the hardware resources through standard subroutines or application programming interfaces (APIs) that may be included in operating system instructions 145. The executed instructions of operating system 145 may then interact directly with these hardware components.

[0053] The face detection engine 144 may include instructions, when executed by the processor 160, to cause the processor 160 to initiate face detection on image data representing one or more objects within the field of view of the image capture device 100. For example, the processor 160 may execute the face detection engine 144 to determine a Region of Interest (ROI) within the field of view of the lens of the camera optics and sensor 115, which includes one or more faces of the corresponding object. In some cases, the face detection engine 144 may obtain raw image sensor data of the image within the field of view of the lens of camera optics and sensor 115A or camera optics and sensor 115B when executed by the processor 160. The executed face detection engine 144 may also initiate face detection and may determine whether one or more faces of an object are in the field of view by, for example, performing face detection operations locally within the processor 160. Face detection operations may include, but are not limited to, performing calculations to determine whether the field of view of the image capture device 100 contains one or more faces, and if so, determining (e.g., identifying) the region (e.g., ROI) containing one or more faces within the FOV.

[0054] The depth determination engine 146 may include instructions, when executed by the processor 160, to cause the processor 160 to determine depth based on image data captured using camera optics and sensor 115A or camera optics and sensor 115B. For example, the processor 160 may execute the depth determination engine 146 to identify features in first image data captured using camera optics and sensor 115A and second image data captured using camera optics and sensor 115B. Furthermore, based on the execution of the depth determination engine 146, the processor 160 generates a disparity map using any techniques known in the art. The disparity map may include depth values ​​identifying the scene depth at each corresponding pixel location (e.g., the pixel location corresponding to each identified feature).

[0055] Image processing instructions 143 may include instructions that, when executed, cause processor 160 to perform one or more image processing operations relating to the captured image data. These image processing operations include, but are not limited to, de-mosaicing, noise reduction, crosstalk reduction, color processing, gamma adjustment, image filtering (e.g., spatial image filtering), lens artifact or defect correction, image sharpening, or other image processing functions. Furthermore, the executed image processing instructions 143 may generate multi-view images, such as 3D images, based on first image data captured using camera optics and sensor 115A, second image data captured using camera optics and sensor 115B, and generated depth values.

[0056] The hinge angle control engine 147 may include instructions that, when executed by the processor 160, cause the processor 160 to adjust the hinge angle, such as the hinge angle of a hinge assembly connecting a first portion 117A of the image capture device 100 to a second portion 117B of the image capture device 100. For example, the processor 160 may execute the hinge angle control engine 147 to control a hinge control 119, which may include a hinge assembly operatively coupled to both the first portion 117A and the second portion 117B. For example, the processor 160 may issue commands to the hinge control 119 to open or close the hinge assembly, thereby increasing or decreasing the hinge angle, respectively.

[0057] The luminance detection engine 149 may include instructions, when executed by the processor 160, to determine a value (e.g., a luminance value) based on the pixel values ​​of pixels in the captured image data and the pixel values ​​of pixels within a detected region of interest (ROI), such as an ROI detected by the processor 160 executing the face detection engine 144. For example, when executed by the processor 160, the luminance detection engine 149 may determine a first value based on the luminance pixel values ​​of all pixels in the captured image (e.g., image data within the field of view of the lens of camera optics and sensor 115A or camera optics and sensor 115B). The executed luminance detection engine 149 may also cause the processor 160 to determine a second value based on the luminance pixel values ​​of all pixels within the detected ROI, including the face of the object. In some examples, one or more of the first and second values ​​include the average luminance pixel value of the corresponding pixel values. In other examples, one or more of the first and second values ​​include the median luminance pixel value of the corresponding pixel values. In other examples, the first and second values ​​may be determined based on any suitable mathematical or statistical process or technique (e.g., but not limited to determining the sum of squares).

[0058] The brightness-based dynamic range detection engine 151 may include instructions, when executed by the processor 160, to determine whether captured image data (e.g., image sensor data) identifies a "high dynamic range" scene or a "non-high dynamic range" scene based on values ​​(e.g., a first value and a second value) determined by the executed brightness detection engine 149. For example, when executed by the processor 160, the executed brightness-based dynamic range detection engine 151 may compare the first value with the second value and determine whether the captured image data identifies a "high dynamic range" scene or a "non-high dynamic range" scene based on the comparison. In some cases, the executed brightness-based dynamic range detection engine 151 may determine the difference between the first value and the second value, and if the difference is greater than a threshold amount (e.g., a predetermined threshold amount), the executed brightness-based dynamic range detection engine 151 may determine that the captured image data identifies a "high dynamic range" scene. Alternatively, if the difference is equal to or less than the threshold amount, the executed brightness-based dynamic range detection engine 151 may determine that the captured image data identifies a "non-high dynamic range" scene. In other cases, the brightness-based dynamic range detection engine 151 can determine whether the captured image data identifies a "high dynamic range" scene or a "non-high dynamic range" scene by applying any suitable mathematical or statistical process or technique to the first and second values.

[0059] The lighting effects engine 153 may include instructions, when executed by the processor 160, to apply one or more lighting effects to image data captured by camera optics and sensor 115A or camera optics and sensor 115B. Examples of lighting effects include color shift, depth of field (e.g., bokeh), AR, VR, and averaging effects. In some examples, the executed lighting effects engine 153 applies one or more lighting effects to a multi-view image generated by the executed image processing engine 143.

[0060] Despite Figure 1 In this document, processor 160 is located within image capture device 100, but in some examples, processor 160 may include one or more cloud-distributed processors. For example, one or more of the functions described herein with respect to processor 160 may be implemented (e.g., executed) by one or more remote processors (such as one or more cloud processors within a corresponding cloud-based server). The cloud processors may communicate with processor 160 via a network, wherein processor 160 is connected to the network via transceiver 111. Each cloud processor may be coupled to a non-transitory cloud storage medium, which may be co-located with or remote from the corresponding cloud processor. The network may be any Personal Area Network (PAN), Local Area Network (LAN), Wide Area Network (WAN), or the Internet.

[0061] Figure 2 It is shown Figure 1 A schematic diagram of exemplary components of the image capture device 100 is shown. As illustrated, a processor 160 is communicatively coupled to each of camera optics and sensors 115A and 115B. For example, the processor 160 can provide an image capture command 201A to cause camera optics and sensors 115A to capture first image data 203A. Similarly, the processor 160 can provide an image capture command 201B to cause camera optics and sensors 115B to capture second image data 203B. Camera optics and sensors 115A may be disposed within (e.g., positioned thereon) a first portion 117A of the image capture device 100, and camera optics and sensors 115B may be disposed within a second portion 117B of the image capture device 100. The first portion 117A may include a first inner surface (e.g., a user-facing surface) of a foldable display (e.g., foldable display 125), and the second portion 117B may include a second inner surface of the foldable display. The image capture device 100 can be folded along the central portion that separates the first part 117A and the second part 117B of the image capture device.

[0062] In some examples, image capture device 100 may include timer 212. In some examples, timer 212 may be an executable timer. Processor 160 may receive timestamp data 213 from timer 212. Processor 160 may associate the timestamp with image data captured from camera optics and sensor 115A and camera optics and sensor 115B. For example, after receiving first image data 203A from camera optics and sensor 115A, processor 160 may request (e.g., read) a timestamp from timer 212. Processor 160 may obtain timestamp data 213 identifying the timestamp and may associate the timestamp with the first image data 203A. For example, processor 160 may include the timestamp as metadata associated with the first image data 203A. Similarly, upon receiving second image data 203B from camera optics and sensor 115B, processor 160 may request and obtain timestamp data 213 from timer 212 and may associate the obtained timestamp with the second image data 203B.

[0063] Furthermore, processor 160 is operatively coupled to hinge control 119. In this example, hinge control 119 includes motor 208, position sensor 210, and hinge assembly 206. Hinge assembly 206 may include one or more hinges connecting a first portion 117A of image capture device 100 to a second portion 117B of image capture device 100. Motor 208 is operatively coupled to hinge assembly 206 and operable to adjust (e.g., open or close) one or more hinges. For example, processor 160 may provide motor command 225 to motor 208. In response, motor 208 may cause hinge assembly 206 to adjust one or more hinges, thereby increasing or decreasing the hinge angle between the first portion 117A and the second portion 117B. For example, motor 208 may rotate in one direction (e.g., clockwise) to open one or more hinges and may rotate in another direction (e.g., counterclockwise) to close one or more hinges.

[0064] Additionally, processor 160 can receive position data 227 from position sensor 210, wherein position data 227 identifies the hinge angle of one or more hinges of hinge assembly 206. In some examples, to adjust the hinge angle, processor 160 obtains position data 227 to determine a first hinge angle. To adjust the hinge angle to a second hinge angle, processor 160 can compare the second hinge angle with the first hinge angle. Based on this comparison, processor 160 can generate and provide motor command 225 to adjust the hinge angle. For example, processor 160 can determine the difference between the second hinge angle and the first hinge angle. Processor 160 can then generate motor command 225 identifying the determined difference and provide motor command 225 to motor 208.

[0065] In some examples, image capture device 100 may adjust the hinge angle based on received user input (such as user input received via input device 170). For example, image capture device 100 may display a graphical user interface (GUI) (e.g., via foldable display 125) that allows the user to provide user input. Based on the user input, processor 160 generates motor commands 225 and provides motor commands 225 to motor 208 to adjust the hinge angle. In some examples, the GUI notifies the user whether to adjust (e.g., open or close) the hinge angle to optimize image quality (e.g., maximize scene, diversify features for multi-view compositing, bokeh, etc.). For example, image capture device 100 may determine the object depth of objects within the captured image data, as described herein. Based on the determined object depth, image capture device 100 may notify the user to provide input via one or more GUI elements to adjust (e.g., open or close) the hinge angle to optimize image quality. In some examples, image capture device 100 outputs GUI elements on a display (such as foldable display 125) based on the adjusted angle.

[0066] Figure 3A , 3B Exemplary camera configurations of each of camera optics and sensor 115A and camera optics and sensor 115B are shown in 3C and 3D diagrams. For example, Figure 3A A single camera 302 for camera optics and sensor 115A is shown, as well as a single camera 304 for camera optics and sensor 115B. Figure 3B Camera optics and sensors 115A with a first camera 302A and a second camera 302B are shown, as well as camera optics and sensors 115B with a first camera 304A and a second camera 304B. Although cameras 302A and 302B, and cameras 304A and 304B are shown horizontally, they can be positioned vertically or in any suitable configuration.

[0067] also, Figure 3C Camera optics and sensors 115A are shown, including a first camera 302A, a second camera 302B, and a third camera 302C. Camera optics and sensors 115B include a first camera 304A, a second camera 304B, and a third camera 304C. Although cameras 302A, 302B, 302C and cameras 304A, 304B, 304C are shown vertically, they can be positioned horizontally or in any suitable configuration. For example, Figure 3D Cameras 302A, 302B, and 302C, which include camera optics and sensor 115A, are shown in a “Y” form (e.g., an inverted triangle). Figure 3DCameras 304A, 304B, and 304C, which also show camera optics and sensors 115B in a similar “Y” form, are also shown.

[0068] Figure 4A , 4B Figures 4C and 4C illustrate a foldable device 400 including an image capture device 100 and a foldable display 125. The foldable device 400 may be, for example, a foldable smartphone or tablet. The foldable device 400 may include a housing having a first portion 401 and a second portion 403. The first portion 401 may be folded relative to the second portion 403. Each of the first portion 401 and the second portion 403 may include one or more displays. For example, as shown, the foldable display 125 includes a first display portion 125A disposed within the first portion 401 and a second display portion 125B disposed within the second portion 403. The first display portions 125A and 125B may be communicatively coupled to a processor 160. Figure 4A , 4B (Not shown in 4C). Each of the first portion 401 and the second portion 403 of the housing may further include one or more camera optics and sensors, such as camera optics and sensors 115A, 115B. Each of the camera optics and sensors 115A and 115B may be included in any suitable configuration (e.g., with respect to...). Figure 3A , 3B Any number of cameras arranged or set up (as shown in the configurations of 3C and 3D).

[0069] In addition, such as Figure 4A As shown, camera optics and sensor 115A can be positioned along the first surface 402 of the first portion 401 of the housing of the foldable device 400, while camera optics and sensor 115B can be positioned along the second surface 404 of the second device portion 403 of the housing of the foldable device 400. Furthermore, as an example, camera optics and sensor 115A can be positioned along a corresponding vertical axis (e.g., Figure 4A The "Y" axis is positioned above the first display portion 125A, and the camera optics and sensor 115B can be positioned above the second display portion 125B along the corresponding vertical axis. In another example, such as Figure 4B As shown, the camera optics and sensor 115A can be positioned below the first display portion 125A along the corresponding vertical axis, and the camera optics and sensor 115B can be positioned below the second display portion 125B along the corresponding vertical axis.

[0070] exist Figure 4CIn the additional example shown, camera optics and sensor 115A can be positioned below the surface of the first display portion 125A, while camera optics and sensor 115B can be positioned below the surface of the second display portion 125B. Since camera optics and sensor 115A and camera optics and sensor 115B are each located below the surface of a corresponding one of the first display portion 125A and the second display portion, light can pass through the first display portion 125A and the second display portion 125B respectively before striking the corresponding sensors. Furthermore, although in Figure 4C In the diagram, each of the camera optics and sensors 115A and 115B is shown as being close to the top portion of the first display portion 125A and the second display portion 125B, respectively. However, in some examples, each of the camera optics and sensors 115A and 115B may be disposed wholly or partially below any additional or alternative portions of the first display portion 125A and the second display portion 125B, respectively.

[0071] The central portion 453 of the foldable device 400 may include one or more hinges, such as 450A, 450B, and 450C, connecting the first portion 401 to the second portion 403. Hinges 450A, 450B, and 450C enable the first portion 401 to fold relative to the second portion 403, thereby allowing the foldable device 400 to open and close. As an example, processor 160 may execute hinge angle control engine 147 to adjust the hinge angle 406 formed between the first inner surface 402 of the first portion 401 and the second inner surface 404 of the second portion 403, based on commands generated and provided to a corresponding motor (e.g., motor 208) connected to one or more of hinges 450A, 450B, or 450C. Additionally, in some examples, an operator or user of the foldable device 400 may manually adjust the hinge angle 406 based on a graphical user interface presented on a portion of display 125.

[0072] Figure 4DA foldable device 400 is shown that captures an image of scene 475. Scene 475 includes, for example, multiple trees 476 and balloons 477. In this example, the camera (such as camera 302) of camera optics and sensor 115A has a focal length 477A relative to the balloons 477 of scene 475. Similarly, the camera of camera optics and sensor 115A has a focal length 477B relative to the balloons 477 of scene 475. Furthermore, at the current hinge angle 406, the camera of camera optics and sensor 115A is separated from the camera of camera optics and sensor 115B by a baseline length 451. The baseline length 451 for each of the plurality of ranges of hinge angle 406 can be determined during calibration of image capture device 100 and can be stored in non-volatile memory (such as storage medium 110). For example, image capture device 100 can determine the current hinge angle 406 as described herein, and can determine the corresponding baseline length 451 (e.g., from camera optics and sensor 115A to camera optics and sensor 115B) based on a table stored in storage medium 110 that maps hinge angle 406 to baseline length 451.

[0073] Processor 160 can execute capture control instructions 135 to capture first image data of scene 475 from camera optics and sensor 115A, and second image data of scene 475 from camera optics and sensor 115A. Furthermore, processor 160 can execute a depth determination engine 146 as described herein to identify features of scene 475 (such as balloon 477) and determine a disparity map based on the first and second image data. Disparity can identify depth values ​​associated with corresponding pixels in the first and second image data. For example, the disparity map may include depth values ​​corresponding to a depth 452 from baseline length 451 to balloon 477.

[0074] Then, processor 160 can adjust hinge angle 406 based on external and internal parameters corresponding to each camera and a disparity map. Processor 160 can adjust hinge angle 406 to align image data captured using camera optics and sensor 115A with image data captured using camera optics and sensor 115B. For example, after adjusting hinge angle 406, processor 160 can execute capture control command 135 to capture third image data of scene 475 from camera optics and sensor 115A and fourth image data of scene 475 from camera optics and sensor 115B. The third image data can be more aligned with the fourth image data than, for example, the alignment of the first image data with the second image data. Processor 160 can execute depth determination engine 146 to generate a second disparity map based on the third and fourth image data.

[0075] Processor 160 can execute image processing instructions 143 to generate multi-view images, such as 3D images, based on third image data, fourth image data, and a second disparity map. In some examples, processor 160 can execute a face detection engine 144 to identify the faces of objects within the generated multi-view images.

[0076] In some examples, processor 160 may execute a lighting engine 153 to apply one or more lighting effects to the generated multi-view image. For example, the executed lighting engine 153 may apply color shift, depth of field (e.g., bokeh effect), AR, VR, averaging effects, or any other image capture enhancement to the multi-view image.

[0077] Figure 5 This is a flowchart of an example process 500 for capturing an image based on an adjusted hinge angle, according to one embodiment. Process 500 can be executed by one or more processors (e.g., processors that execute instructions locally at the image capture device). Figure 1 The image capture device 100 is executed by its processor 160. Therefore, the various operations of process 500 can be represented by executable instructions stored in a storage medium (e.g., storage medium 110 of the image capture device 100) on one or more computing platforms.

[0078] Referring to block 502, the image capture device 100 can acquire first image data from a first sensor, such as image data from camera optics and sensor 115A. The first sensor can be positioned on a first inner surface of the foldable display. For example, camera optics and sensor 115A can be positioned on the surface of the first display portion 125A of the foldable display 125. At block 504, the image capture device 100 can acquire second image data from a second image sensor, such as image data from camera optics and sensor 115B. The second sensor is positioned on a second inner surface of the foldable display. For example, camera optics and sensor 115B can be positioned on the surface of the second display portion 125B of the foldable display 125.

[0079] At box 506, image capture device 100 can determine the depth of an object based on first image data and second image data. For example, and as described herein, image capture device 100 can identify features (e.g., balloon 477) in each of the first and second image data. According to some examples, image capture device 100 matches pixels with key terms in the first image data with pixels with corresponding key terms in the second image data. Image capture device 100 then uses any techniques known in the art to generate a disparity map. The disparity map can indicate the scene depth at each corresponding pixel location. Image capture device 100 can calculate depth values ​​based on the focal length of each camera (e.g., 477 Å), baseline length (e.g., baseline length 45 Å), and the disparity map.

[0080] Upon reaching frame 508, the image capture device 100 adjusts the hinge angle of the hinge connecting the first inner surface to the second inner surface based on the determined depth. For example, the image capture device 100 may increase or decrease the hinge angle based on the determined depth, such as... Figure 4A The hinge angle is 406. At frame 510, the image capture device 100 performs an image capture operation in response to adjustment. For example, the image capture device 100 can acquire third image data from a first sensor and fourth image data from a second sensor. The image capture device 100 can identify corresponding features within the third and fourth image data and generate a second disparity map based on the identified features. The image capture device 100 can then generate a multi-view image based on the third image data, the fourth image data, and the second disparity map.

[0081] Figure 6 This is a flowchart of an example process 600 for performing an image capture operation according to one embodiment. Process 600 may be performed by one or more processors (e.g., processors that execute instructions locally at the image capture device). Figure 1 The image capture device 100 is executed by its processor 160. Therefore, the various operations of process 600 can be represented by executable instructions stored in a storage medium (e.g., storage medium 110 of the image capture device 100) on one or more computing platforms.

[0082] Referring to frame 602, image capture device 100 can determine hinge angles (such as hinge angle 406) of hinges (such as hinge 450A). The hinge connects a first inner surface (e.g., first inner surface 402) of a foldable device (e.g., foldable device 400) having a first sensor (e.g., camera optics and sensor 115A) to a second inner surface (e.g., second inner surface 404) of the foldable device having a second sensor (e.g., camera optics and sensor 115B). For example, image capture device 100 can obtain position data 227 from position sensor 210 to determine, for example, the hinge angle.

[0083] At block 604, the image capture device 100 can determine parameter values ​​corresponding to each of the first and second sensors based on the hinge angle. For example, storage medium 110 can store internal parameters 167 and external parameters 169 for each of the sensors. Furthermore, the internal parameters 167 and external parameters 169 can include parameters for multiple hinge angle ranges. For example, a first set of internal parameters 167 and external parameters 169 can correspond to a first hinge angle range (e.g., from 0 degrees to 5 degrees), a second set of internal parameters 167 and external parameters 169 can correspond to a first hinge angle range (e.g., from 5 degrees to 10 degrees), and so on. In some examples, each hinge angle has corresponding internal and external parameters. The image capture device 100 can obtain internal parameters 167 and external parameters 169 corresponding to the determined hinge angle for each of the first and second sensors.

[0084] Proceeding to block 606, the image capture device 100 can acquire first image data from a first sensor and second image data from a second sensor. For example, the image capture device 100 can acquire image data 203A from camera optics and sensor 115A and image data 203B from camera optics and sensor 115B. At block 608, the image capture device 100 can determine the disparity of an object based on the first image data, the second image data, and parameters. For example, the image capture device 100 can identify objects within the first and second image data and can generate a disparity map including the disparity values ​​of the objects based on external and internal parameters corresponding to each camera, as described herein. The image capture device 100 can apply any known technique to the disparity values ​​to determine the disparity of the object. For example, the image capture device 100 can determine the average value of the disparity values ​​to determine the disparity of the object.

[0085] At block 610, image capture device 100 may determine whether the determined parallax is less than a threshold. For example, the threshold parallax may include a predetermined threshold that image capture device 100 can store in storage medium 110. If the parallax is not less than the threshold, the method proceeds to block 612, and image capture device 100 may adjust the hinge angle. For example, image capture device 100 may provide motor command 225 to motor 208 to adjust the hinge angle (e.g., hinge angle 406) of hinge assembly 206 (e.g., 450A). Method 600 then returns to block 602.

[0086] Returning to reference box 610, if the determined disparity is less than a threshold disparity, the method proceeds to box 614, where the image capture device 100 performs an image capture operation using a first image sensor and a second image sensor. For example, the image capture device 100 may obtain third image data from the first sensor and fourth image data from the second sensor. The image capture device 100 may determine a disparity map based on the third and fourth image data (and in some examples, corresponding camera parameters), and may use any of the procedures described herein to generate a multi-view image based on the third image data, the fourth image data, and the disparity map.

[0087] Figure 7 This is a flowchart of an example process 700 for generating a user-synthesized image according to one embodiment. Process 700 may be executed by one or more processors (e.g., processors that execute instructions locally at the image capture device). Figure 1 The image capture device 100 is executed by its processor 160. Therefore, the various operations of process 700 can be represented by executable instructions stored in a storage medium (e.g., storage medium 110 of the image capture device 100) on one or more computing platforms.

[0088] At block 702, the image capture device 100 can receive input for capturing an image of a scene. For example, a user can provide input (e.g., via I / O 170) for capturing an image of a scene (e.g., scene 475). At block 704, in response to this input, the image capture device 100 obtains first image data from a first sensor located within a first inner surface of the foldable display. The image capture device also obtains second image data from a second sensor located within a second inner surface of the foldable display. For example, camera optics and sensor 115A can be positioned within a first display portion 125A of the display 125, and camera optics and sensor 115B can be positioned within a second display portion 125B of the display 125.

[0089] Proceeding to block 706, the image capture device 100 can detect the face of an object based on the first image data and the second image data. For example, the image capture device 100 can detect the face of an object within the ROI of the first and second image data. Furthermore, at block 708, the image capture device 100 can adjust the hinge angle based on the detected face. The hinge angle (e.g., hinge angle 406) connects the first inner surface of the foldable display to the second inner surface of the foldable display. For example, the image capture device 100 can determine the parallax value of the face and can calculate the depth value of the face based on the focal length of the first sensor, the baseline length between the first and second sensors, and the parallax value. The image capture device can then adjust the hinge angle based on the depth value.

[0090] At box 710, the image capture device 100 can capture images using a first sensor and a second sensor in response to hinge angle adjustment. For example, the image capture device 100 can obtain third image data from the first sensor and fourth image data from the fourth sensor. The image capture device 100 can determine a disparity map based on the third and fourth image data, and generate a multi-view image based on the disparity map, the third image data, and the fourth image data, as described herein.

[0091] Proceeding to block 712, the image capture device 100 may apply lighting effects to the captured image to generate a user-synthesized image. For example, the image capture device 100 may apply any lighting effect (such as bokeh effect) selected by the user to the multi-view image to generate a user-synthesized image. At block 714, the image capture device 100 may store the user-synthesized image in a corresponding tangible non-transitory memory (such as storage medium 110).

[0092] Examples of implementation methods are further described in the following numbered clauses:

[0093] 1. An image capture device, comprising:

[0094] The housing includes a first housing portion and a second housing portion;

[0095] Non-transitory machine-readable storage medium for storing instructions; and

[0096] At least one processor, coupled to the non-transitory machine-readable storage medium, is configured to execute the instructions to:

[0097] First image data is obtained from a first sensor coupled to the first housing portion of the housing;

[0098] Second image data is obtained from a second sensor coupled to the second housing portion of the housing;

[0099] The depth of the object is determined based on the first image data and the second image data;

[0100] The adjusted angle between the first housing portion and the second housing portion is output based on the depth; and an image capture operation is performed based on the adjusted angle.

[0101] 2. The image capture device according to Clause 1, wherein the at least one processor is further configured to execute the instructions to:

[0102] Based on the output of the adjusted angle, third image data is obtained from the first sensor and fourth image data is obtained from the second sensor; and

[0103] A multi-view image is generated based on the data from the third sensor and the data from the fourth sensor.

[0104] 3. The image capture device according to Clause 2, wherein the at least one processor is further configured to execute the instructions to:

[0105] User-synthesized images are generated by applying optical effects to the multi-view images; and

[0106] The user-synthesized image is stored in the non-transitory machine-readable storage medium.

[0107] 4. The image capture device according to Clause 3, wherein the at least one processor is further configured to execute the instructions to:

[0108] Segmenting the multi-view image; and

[0109] The light effect is applied based on the segmented multi-view image.

[0110] 5. The image capturing device according to Clause 4, wherein the light effect includes bokeh effect.

[0111] 6. The image capture device according to any one of clauses 1-5, wherein:

[0112] The object includes a face; and

[0113] The at least one processor is also configured to execute the instructions to detect the face.

[0114] 7. The image capture device according to any one of clauses 1-6, wherein the at least one processor is further configured to execute the instructions to:

[0115] Determine the angle between the first housing portion and the second housing portion; and

[0116] The depth of the object is determined based on the angle.

[0117] 8. The image capture device according to any one of clauses 1-7, wherein the at least one processor is further configured to execute the instructions to generate parallax data based on the first image data and the second image data.

[0118] 9. The image capture device according to Clause 8, wherein the at least one processor is further configured to execute the instructions to:

[0119] Determine the angle between the first portion and the second portion of the outer casing;

[0120] The baseline length between the first sensor and the second sensor is determined based on the determined angle;

[0121] Determine the focal length of at least one of the first sensor or the second sensor; and

[0122] Depth values ​​are generated based on the parallax data, the baseline length, and the focal length.

[0123] 10. The image capture device according to any one of clauses 8-9, wherein the at least one processor is further configured to execute the instructions to:

[0124] Determine the disparity of the object based on the disparity data; and

[0125] The adjusted angle is output based on the disparity and disparity threshold.

[0126] 11. The image capture device according to any one of clauses 1-10, wherein the at least one processor is further configured to execute the instructions to:

[0127] Obtain position data from position sensors;

[0128] Based on the position data, determine the angle between the first housing portion and the second housing portion; and

[0129] The adjusted angle is determined based on the determined angle.

[0130] 12. The image capture device according to any one of clauses 1-11, wherein the at least one processor is further configured to execute the instructions to:

[0131] Determine the angle between the first housing portion and the second housing portion;

[0132] Determine one or more camera parameters based on the determined angle; and

[0133] The adjusted angle is output based on the one or more camera parameters.

[0134] 13. The image capture device according to claim 12, wherein:

[0135] The one or more camera parameters include a translation matrix; and

[0136] The at least one processor is further configured to execute the instructions to translate the reference of the first sensor to the corresponding reference of the second sensor based on the translation matrix.

[0137] 14. The image capture device according to any one of clauses 1-13, wherein:

[0138] The first housing portion includes a first portion of the display;

[0139] The second housing portion includes the second portion of the display; and

[0140] The first sensor is disposed below the first portion of the display, and the second sensor is disposed below the second portion of the display.

[0141] 15. The image capture device according to any one of Clauses 1-14 further includes a motor.

[0142] 16. The image capture device according to Clause 15, wherein the motor is configured to adjust the angle between the first housing portion and the second housing portion based on the adjusted angle.

[0143] 17. The image capture device according to any one of clauses 1-16, wherein the processor is further configured to execute the instructions to output GUI elements on the display based on the adjusted angle.

[0144] 18. A method for operating an image capture device, comprising:

[0145] First image data is obtained from a first sensor coupled to a first portion of the housing of the image capture device;

[0146] Second image data is obtained from a second sensor coupled to a second portion of the housing;

[0147] The depth of the object is determined based on the first image data and the second image data;

[0148] The adjusted angle between the first housing portion and the second housing portion is output based on the determined depth; and

[0149] An image capture operation is performed in response to the adjusted angle.

[0150] 19. The method according to Clause 18, wherein performing the image capture operation comprises:

[0151] Based on the output of the adjusted angle, third image data is obtained from the first sensor and fourth image data is obtained from the second sensor; and

[0152] A multi-view image is generated based on the data from the third sensor and the data from the fourth sensor.

[0153] 20. The method described under Clause 19 further includes:

[0154] User-synthesized images are generated by applying optical effects to the multi-view images; and

[0155] The user-generated image is stored in a memory device.

[0156] 21. The method described under Clause 20 further includes:

[0157] Segmenting the multi-view image; and

[0158] The light effect is applied based on the segmented multi-view image.

[0159] 22. The method according to Clause 21, wherein the light effect includes bokeh effect.

[0160] 23. The method according to any one of clauses 18-22, wherein the object includes a face, and the method further includes detecting the face.

[0161] 24. The method according to any one of clauses 18-23 further includes:

[0162] Determine the angle between the first housing portion and the second housing portion of the housing; and

[0163] The depth of the object is determined based on the angle.

[0164] 25. The method according to any one of aspects 18-22, wherein determining the depth of the object comprises: generating parallax data based on the first image data and the second image data.

[0165] 26. The method according to Clause 25, wherein determining the depth of the object comprises:

[0166] Determine the angle between the first housing portion and the second housing portion;

[0167] The baseline length between the first sensor and the second sensor is determined based on the determined angle;

[0168] Determine the focal length of at least one of the first sensor or the second sensor; and

[0169] Depth values ​​are generated based on the parallax data, the baseline length, and the focal length.

[0170] 27. The method according to any one of clauses 25-26 further includes:

[0171] Determine the disparity of the object based on the disparity data; and

[0172] The adjusted angle is output based on the disparity and disparity threshold.

[0173] 28. The method according to any one of clauses 18-27 further includes:

[0174] Obtain position data from position sensors;

[0175] Determine the angle between the first housing portion and the second housing portion; and

[0176] The adjusted angle is determined based on the determined angle.

[0177] 29. The method described under Clauses 18-28 further includes:

[0178] Determine the angle between the first housing portion and the second housing portion;

[0179] Determine one or more camera parameters based on the determined angle; and

[0180] The adjusted angle is output based on the one or more camera parameters.

[0181] 30. The method described according to Clause 29, wherein:

[0182] The one or more camera parameters include a translation matrix; and

[0183] The method further includes: translating the reference of the first sensor to the corresponding reference of the second sensor based on the translation matrix.

[0184] 31. The method according to any one of clauses 18-30 further includes: outputting GUI elements on the display based on the adjusted angle.

[0185] 32. A non-transitory machine-readable storage medium for storing instructions, said instructions causing said at least one processor to perform an operation when executed by said at least one processor, said operation comprising:

[0186] First image data is acquired from a first sensor in a first housing portion coupled to the housing of the image capture device.

[0187] Second image data is obtained from a second sensor coupled to a second housing portion of the housing;

[0188] The depth of the object is determined based on the first image data and the second image data;

[0189] The adjusted angle between the first and second portions is output based on the depth; and

[0190] The image capture operation is performed based on the adjusted angle.

[0191] 33. The non-transitory machine-readable storage medium according to clause 32, wherein performing the at least one image capture operation includes:

[0192] Based on the output of the adjusted angle, third image data is obtained from the first sensor and fourth image data is obtained from the second sensor; and

[0193] A multi-view image is generated based on the data from the third sensor and the data from the fourth sensor.

[0194] 34. The non-transitory machine-readable storage medium according to clause 33, wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform further operations, the further operations including:

[0195] User-synthesized images are generated by applying optical effects to the multi-view images; and

[0196] The user-generated image is stored in a memory device.

[0197] 35. The non-transitory machine-readable storage medium according to clause 34, wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform further operations, the further operations including:

[0198] Segmenting the multi-view image; and

[0199] The light effect is applied based on the segmented multi-view image.

[0200] 36. The non-transitory machine-readable storage medium as described in Clause 35, wherein the optical effect includes bokeh effect.

[0201] 37. A non-transitory machine-readable storage medium according to any one of clauses 32-36, wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform further operations, the further operations including detecting a face, wherein the object includes the detected face.

[0202] 38. A non-transitory machine-readable storage medium according to any one of clauses 32-37, wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform further operations, the further operations including:

[0203] Determine the angle between the first housing portion and the second housing portion of the housing; and

[0204] The depth of the object is determined based on the angle.

[0205] 39. A non-transitory machine-readable storage medium according to any one of clauses 32-38, wherein determining the depth of the object comprises: generating parallax data based on the first image data and the second image data.

[0206] 40. The non-transitory machine-readable storage medium as described in Clause 39, wherein determining the depth of the object includes:

[0207] Determine the angle between the first housing portion and the second housing portion;

[0208] The baseline length between the first sensor and the second sensor is determined based on the determined angle;

[0209] Determine the focal length of at least one of the first sensor or the second sensor; and

[0210] Depth values ​​are generated based on the parallax data, the baseline length, and the focal length.

[0211] 41. A non-transitory machine-readable storage medium according to any one of clauses 39 and 40, wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform further operations, the further operations including:

[0212] The disparity of the object is determined based on the disparity data; and...

[0213] The adjusted angle output between the first part and the second part is based on the disparity and the disparity threshold.

[0214] 42. A non-transitory machine-readable storage medium according to any one of clauses 32-41, wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform further operations, the further operations including:

[0215] Obtain position data from position sensors;

[0216] Determine the angle between the first housing portion and the second housing portion; and

[0217] The adjusted angle is determined based on the determined angle.

[0218] 43. The non-transitory machine-readable storage medium according to any one of clauses 32-42, wherein determining the depth of the object further includes:

[0219] Determine the angle between the first housing portion and the second housing portion;

[0220] One or more camera parameters are determined based on the determined angle, wherein the adjusted angle is output based on the one or more camera parameters; and

[0221] The adjusted angle is output based on the one or more camera parameters.

[0222] 44. The non-transitory machine-readable storage medium according to Clause 43, wherein the one or more camera parameters include a translation matrix, and wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform further operations, the further operations including: translating a reference of the first sensor to a corresponding reference of the second sensor based on the translation matrix.

[0223] 45. A non-transitory machine-readable storage medium according to any one of clauses 32-44, wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform further operations, the further operations including: outputting GUI elements on a display based on the adjusted angle.

[0224] 46. ​​A system comprising:

[0225] A unit for acquiring first image data from a first sensor in a first housing portion coupled to the housing of an image capture device;

[0226] A unit for obtaining second image data from a second sensor coupled to a second housing portion of the housing;

[0227] A unit for determining the depth of an object based on the first image data and the second image data;

[0228] A unit for outputting an adjusted angle between the first portion and the second portion based on the depth; and

[0229] A unit for performing image capture operations based on the adjusted angle.

[0230] 47. The system according to clause 46, wherein the unit for performing the at least one image capture operation comprises:

[0231] A unit for obtaining third image data from the first sensor and fourth image data from the second sensor based on the output of the adjusted angle; and

[0232] A unit for generating multi-view images based on the data from the third sensor and the data from the fourth sensor.

[0233] 48. The system described in Clause 47 further includes:

[0234] A unit for generating a user-synthesized image based on applying light effects to the multi-view image; and

[0235] A unit for storing the user-synthesized image in a memory device.

[0236] 49. The system described in Clause 48 further includes:

[0237] Units for segmenting the multi-view image; and

[0238] A unit for applying the light effect based on the segmented multi-view image.

[0239] 50. The system according to Clause 49, wherein the light effect includes bokeh effect.

[0240] 51. The system according to any one of clauses 46-50 further includes: a unit for detecting a face, wherein the object includes the face.

[0241] 52. The system according to any one of clauses 46-51 further includes:

[0242] A unit for determining the angle between the first housing portion and the second housing portion of the housing; and

[0243] A unit for determining the depth of the object based on the angle.

[0244] 53. The system according to any one of clauses 46-52, wherein the unit for determining the depth of the object comprises: a unit for generating parallax data based on the first image data and the second image data.

[0245] 54. The system according to clause 53, wherein the unit for determining the depth of the object comprises:

[0246] A unit for determining the angle between the first housing portion and the second housing portion;

[0247] A unit for determining the baseline length between the first sensor and the second sensor based on the determined angle;

[0248] A unit for determining the focal length of at least one of the first sensor or the second sensor; and

[0249] A unit for generating depth values ​​based on the disparity data, the baseline length, and the focal length.

[0250] 55. The system according to any one of clauses 53 and 54 further includes:

[0251] A unit for determining the disparity of the object based on the disparity map; and

[0252] A unit for outputting the adjusted angle based on the disparity and the disparity threshold.

[0253] 56. The system according to any one of clauses 46-55 further includes:

[0254] A unit used to obtain position data from a position sensor;

[0255] A unit for determining the angle between the first housing portion and the second housing portion; and

[0256] A unit for determining the adjusted angle based on the determined angle.

[0257] 57. The system according to any one of clauses 46-55 further includes:

[0258] A unit for determining the angle between the first housing portion and the second housing portion; and

[0259] A unit for determining one or more camera parameters based on a determined angle; and

[0260] A unit for outputting the adjusted angle based on the one or more camera parameters.

[0261] 58. The system described in Clause 57, wherein:

[0262] The one or more camera parameters include a translation matrix; and

[0263] The system further includes a unit for translating a reference of the first sensor to a corresponding reference of the second sensor based on the translation matrix.

[0264] 59. The system according to any one of clauses 46-58, wherein:

[0265] The first housing portion includes a first portion of the display;

[0266] The second housing portion includes the second portion of the display; and

[0267] The first sensor is disposed below the first portion of the display, and the second sensor is disposed below the second portion of the display.

[0268] 60. The system according to any one of clauses 46-59 further includes: a unit for adjusting the angle between the first housing portion and the second housing portion based on the adjusted angle.

[0269] 61. The system according to any one of clauses 46-60 further includes: a unit for outputting GUI elements on a display based on the adjusted angle.

[0270] Although the method described above is based on the illustrated flowchart, many other ways of performing the actions associated with the method can be used. For example, the order of some operations can be changed, and some embodiments may omit one or more of the described operations and / or include additional operations.

[0271] Furthermore, the methods and systems described herein can be embodied, at least in part, as computer-implemented processes and apparatus for performing those processes. The disclosed methods can also be embodied, at least in part, as a tangible, non-transitory machine-readable storage medium encoded with computer program code. For example, the methods can be embodied as hardware, executable instructions (e.g., software) executed by a processor, or a combination of both. The medium can include, for example, RAM, ROM, CD-ROM, DVD-ROM, BD-ROM, hard disk drive, flash memory, or any other non-transitory machine-readable storage medium. When the computer program code is loaded into and executed by the computer, the computer becomes an apparatus for performing the methods. The methods can also be embodied, at least in part, as a computer, with the computer program code loaded into or executed in the computer, making the computer a dedicated computer for performing the methods. When implemented on a general-purpose processor, computer program code segments configure the processor to create specific logic circuits. Alternatively, the methods can be embodied, at least in part, as application-specific integrated circuits (ASICs) for performing the methods.

Claims

1. An image capture device, comprising: The housing includes a first housing portion and a second housing portion; Non-transitory machine-readable storage medium for storing instructions; as well as At least one processor, coupled to the non-transitory machine-readable storage medium, is configured to execute the instructions to: First image data is obtained from a first sensor coupled to the first housing portion of the housing; Second image data is obtained from a second sensor coupled to the second housing portion of the housing; Determine the angle between the first housing portion and the second housing portion; One or more external parameters are determined based on the determined angle, the one or more external parameters including at least one of a first translation matrix associated with the first sensor or a second translation matrix associated with the second sensor; The first image data and the second image data are aligned by translating the reference of the first sensor to the corresponding reference of the second sensor based on at least one of the first translation matrix or the second translation matrix. The depth of the object is determined based on the aligned first and second image data; The adjusted angle between the first housing portion and the second housing portion is output based on the depth. as well as The image capture operation is performed based on the adjusted angle.

2. The image capture device according to claim 1, wherein, The at least one processor is further configured to execute the instructions to: Based on the output of the adjusted angle, third image data is obtained from the first sensor and fourth image data is obtained from the second sensor; and A multi-view image is generated based on the third image data and the fourth image data.

3. The image capture device according to claim 2, wherein, The at least one processor is further configured to execute the instructions to: User-synthesized images are generated by applying optical effects to the multi-view images; and The user-synthesized image is stored in the non-transitory machine-readable storage medium.

4. The image capture device according to claim 3, wherein the at least one processor is further configured to execute the instructions to: Segmenting the multi-view image; and The light effect is applied based on the segmented multi-view image.

5. The image capture device according to claim 4, wherein, The light effects include bokeh.

6. The image capture device according to claim 1, wherein: The object includes a face; and The at least one processor is also configured to execute the instructions to detect the face.

7. The image capture device according to claim 1, wherein, The at least one processor is also configured to execute the instructions to further determine the depth of the object based on the determined angle.

8. The image capture device according to claim 1, wherein, The at least one processor is further configured to execute the instructions to generate parallax data based on the first image data and the second image data.

9. The image capture device according to claim 8, wherein, The at least one processor is further configured to execute the instructions to: The baseline length between the first sensor and the second sensor is determined based on the determined angle; Determine the focal length of at least one of the first sensor or the second sensor; as well as Depth values ​​are generated based on the parallax data, the baseline length, and the focal length.

10. The image capture device according to claim 8, wherein, The at least one processor is further configured to execute the instructions to: Determine the disparity of the object based on the disparity data; and The adjusted angle is output based on the disparity and disparity threshold.

11. The image capture device according to claim 1, wherein, The at least one processor is further configured to execute the instructions to: Obtain position data from position sensors; The angle between the first housing portion and the second housing portion is determined based on the position data; as well as The adjusted angle is further determined based on the determined angle.

12. The image capture device according to claim 1, wherein: The first housing portion includes a first portion of the display; The second housing portion includes the second portion of the display; as well as The first sensor is disposed below the first portion of the display, and the second sensor is disposed below the second portion of the display.

13. The image capture device according to claim 1, further comprising a motor.

14. The image capture device according to claim 13, wherein, The motor is configured to adjust the angle between the first housing portion and the second housing portion based on the adjusted angle.

15. The image capture device according to claim 1, wherein, The at least one processor is also configured to execute the instructions to output GUI elements on the display based on the adjusted angle.

16. A method for operating an image capture device, comprising: First image data is obtained from a first sensor in a first housing portion coupled to the housing of the image capture device; Second image data is obtained from a second sensor coupled to a second housing portion of the housing; Determine the angle between the first housing portion and the second housing portion; One or more external parameters are determined based on the determined angle, the one or more external parameters including at least one of a first translation matrix associated with the first sensor or a second translation matrix associated with the second sensor; The first image data and the second image data are aligned by translating the reference of the first sensor to the corresponding reference of the second sensor based on at least one of the first translation matrix or the second translation matrix. The depth of the object is determined based on the aligned first and second image data; The adjusted angle between the first housing portion and the second housing portion is output based on the determined depth. as well as The image capture operation is performed based on the adjusted angle.

17. The method according to claim 16, wherein, Performing the image capture operation includes: Based on the output of the adjusted angle, third image data is obtained from the first sensor and fourth image data is obtained from the second sensor; and A multi-view image is generated based on the third image data and the fourth image data.

18. The method of claim 17, further comprising: User-synthesized images are generated by applying light effects to the multi-view images; as well as The user-generated image is stored in a memory device.

19. The method according to claim 18, wherein, Generating the user-synthesized image includes: Segmenting the multi-view image; and The optical effect is applied based on the segmentation.

20. The method according to claim 19, wherein, The light effects include bokeh.

21. The method according to claim 16, wherein, The object includes a face, and the method further includes detecting the face.

22. The method of claim 16, further comprising: The depth of the object is further determined based on the determined angle.

23. The method according to claim 16, wherein, Determining the depth of the object includes generating disparity data based on the first image data and the second image data.

24. The method according to claim 23, wherein, Determining the depth of the object includes: The baseline length between the first sensor and the second sensor is determined based on the determined angle; Determine the focal length of at least one of the first sensor or the second sensor; and Depth values ​​are generated based on the parallax data, the baseline length, and the focal length.

25. The method of claim 23, further comprising: The disparity of the object is determined based on the disparity data; as well as The adjusted angle is output based on the disparity and disparity threshold.

26. The method of claim 16, further comprising: Obtain position data from position sensors; The angle between the first housing portion and the second housing portion is determined based on the position data; as well as The adjusted angle is further determined based on the determined angle.

27. The method of claim 16, wherein: The first housing portion includes a first portion of the display; The second housing portion includes the second portion of the display; as well as The first sensor is disposed below the first portion of the display, and the second sensor is disposed below the second portion of the display.

28. The method of claim 16, further comprising: Using a motor, the angle between the first housing portion and the second housing portion is adjusted based on the adjusted angle.

29. The method of claim 28, further comprising: The GUI elements are displayed based on the adjusted angle.

Images