Learning transformation matrices for multi-sensor image capture devices

Abstract

Artifacts in a sequence of image frames may be reduced or eliminated by modification to align an input image frame with another image frame in the sequence, such as by geometrically warping to generate a corrected image frame having a field of view aligned with another frame in the sequence of frames having image frames. The warping may be performed based on a model generated from data relating to a multi-sensor device. Disparity between image frames may be modeled based on image capture from a first and second image sensor for a scene at various depths. The model may be used to predict disparity values ​​of captured images, and those predicted disparity values ​​may be used to reduce artifacts resulting from image sensor switching. The predicted disparity values ​​may be used in image conditions that result in erroneous actual disparity values.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. patent application Ser. No. 17 / 201,660, filed Mar. 15, 2021, entitled “TRANSFORM MATRIX LEARNING FOR MULTI-SENSOR IMAGE CAPTURE DEVICES,” which is expressly incorporated by reference herein in its entirety.

[0002] FIELD OF THE DISCLOSURE Aspects of the present disclosure relate generally to image processing. Several features of the present disclosure can enable and provide improved processing of images acquired from a multi-sensor image capture device. [Background technology]

[0003] Image capture devices, devices capable of capturing one or more digital images, whether still image photographs or a sequence of images for a video, may be incorporated into a wide variety of devices. By way of example, image capture devices may include standalone digital cameras or digital video camcorders, mobile phones, cellular or satellite radio telephones, personal digital assistants (PDAs), panels or tablets, camera-equipped wireless communication device handsets such as gaming devices, computing devices such as webcams, video surveillance cameras, or other devices with digital imaging or video capabilities.

[0004] Some image capture devices include multiple image sensors that capture image data through one or more lenses, and such devices are referred to as multi-sensor image capture devices or multi-sensor devices. Multiple image sensors may be configured with different lenses to provide multiple fields of view of a scene and / or different zoom levels of a scene. Exemplary lens types include a wide-angle lens, an ultra-wide-angle lens, a telephoto lens, a telescope lens, a periscope zoom lens, a fisheye lens, a macro lens, a prime lens, or various combinations thereof. In one example, a dual camera configuration may include both a wide-angle lens and a telephoto lens.

[0005] However, the use of multiple image sensors increases the complexity of image processing in the device, since users are generally not interested in multiple images of a scene, but rather in capturing and displaying a single image. Thus, multiple frames captured from multiple image sensors may be processed to generate a single image for the user. Nevertheless, and further, due to different physical characteristics between image sensors, frames acquired from each image sensor may flow together in such a way that a transition from one image sensor to another is perceptible to the human eye. For example, changing from a telephoto lens to a wide-angle lens while capturing a series of image frames for display as a video may result in the appearance of a scene shift in the video. Artifacts such as scene shifts in the output of a multi-sensor image capture device are undesirable. Summary of the Invention [Means for solving the problem]

[0006] Artifacts in video frames result in a disruption of the continuity of image frames output from a multi-sensor device due to switching from one image sensor to another image sensor having a different field of view. The artifacts may be reduced or eliminated by modification of an input image frame to match another image frame. For example, an input frame may be geometrically warped by stretching, shrinking, cropping, and / or otherwise modified to generate a corrected image frame having a field of view that is aligned with another frame in a sequence of frames having image frames. The image frame modification may be performed based on a model generated from data regarding the multi-sensor device.

[0007] The warping may be based on a shift between the field of view from the first image sensor to the second image sensor, called disparity. The disparity may be modeled based on image capture from the first and second image sensors for a scene at various depths. The model may be used to predict disparity values ​​for the captured images, and those predicted disparity values ​​were used to reduce artifacts resulting from image sensor switching. The predicted disparity values ​​may be used in image conditions that result in erroneous actual disparity values. For example, if the actual disparity values ​​are determined to be unavailable or erroneous, the predicted disparity values ​​may substitute for the actual disparity values. This substitution may reduce artifacts due to failures of image sensor switching and / or other components associated with image capture, such as computer vision processing (CVP). The predicted disparity values ​​determined from the model may facilitate switching image sensors at any zoom ratio and for any scene, since predicted disparity values ​​are available in conditions where actual disparity determination may fail.

[0008] In some embodiments, a single model may be used to model the disparity of all image sensors. In some embodiments, a separate model may be used for each pair of image frames captured from two image sensors. For example, on a device with three image sensors, a first model may be determined corresponding to the first and second image sensors, a second model may be determined for the first and third image sensors, and a third model may be determined for the second and third image sensors. Thus, switching from any one image sensor to any other image sensor may have predicted disparity values ​​available for warping image frames from one image sensor to align with the other image sensor.

[0009] The following summarizes some aspects of the present disclosure in order to provide a basic understanding of the described technology. This summary is not an extensive overview of all contemplated features of the present disclosure, nor does it identify key or critical elements of all aspects of the present disclosure, nor does it delineate the scope of any or all aspects of the present disclosure. Its sole purpose is to present some concepts of one or more aspects of the present disclosure in summary form as a prelude to the more detailed description that is presented later.

[0010] In general, this disclosure describes image processing techniques involving a digital camera having an image sensor and an image signal processor (ISP). To achieve improved image quality with reduced artifacts during image sensor switching, the ISP may warp image frames received from the image sensor using a disparity model. Warping and disparity modeling may alternatively be performed on image frames stored separately from the image sensor. Warping and disparity modeling may be performed based on a request from another component in the device, such as by the CPU in response to image processing functions running on the CPU.

[0011] The image signal processor may be configured to control the capture of image frames from one or more image sensors and process the image frames from the one or more image sensors to generate views of the scene in corrected image frames. In one example, the image signal processor may receive instructions to capture a sequence of image frames in response to loading of software, such as a camera application, on the CPU. The image signal processor may be configured to generate a single flow of output frames based on each corrected image from the image sensor. The single flow of output frames may include image frames that include image data from the image sensor that have been corrected, such as by geometric warping, to align the image frames with other image frames in the output flow (e.g., previous frames captured by a different image sensor).

[0012] After the output frames representing the scene are generated by the image signal processor, the view of the scene may be displayed on a device display, saved to a storage device as a picture or a sequence of pictures as a video, transmitted over a network, and / or printed on an output medium. For example, the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from different image sensors and then generate corresponding output frames of image data (e.g., preview display frames, still image capture, frames of video, etc.). In other examples, the image signal processor may output frames of image data to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization, generating a video file via the output frames, composing frames for display, composing frames for storage, etc. That is, the image signal processor may obtain incoming frames from one or more image sensors, each coupled to one or more camera lenses, and then generate and output a flow of output frames to various output destinations. In such examples, the image signal processor may be configured to generate a flow of output frames that may have reduced artifacts due to image sensor modifications. In one example, the image signal processor may receive inputs for changing zoom levels based on pinch-zoom actions, gesture detection, or other user inputs to a device including an image sensor or to a user device coupled to a device including an image sensor. When the image sensor changes in response to a user input, such as a pinch-zoom action, the predicted disparity may be used to warp the image frames and reduce artifacts in the sequence of image frames output from the sensor.

[0013] In one aspect of the disclosure, a method includes receiving a first disparity value indicating a difference in field of view between a first image sensor and a second image sensor, receiving a first depth value corresponding to the first disparity value, determining a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, the model being based at least in part on the first disparity value and the first depth, receiving an input image frame from one of the first image sensor or the second image sensor, receiving an input image depth corresponding to the input image frame, determining a predicted disparity value corresponding to the input image depth based at least in part on the model, and / or determining a corrected image frame based at least in part on the input image frame and the predicted disparity value. The corrected image frame may be determined as warped to match a field of view of the other of the first image sensor or the second image sensor based on the predicted disparity value. The predicted disparity values ​​may include values ​​for two or more axes, such that a first disparity value indicates a difference in field of view along a first axis, and the method further includes receiving a second disparity value indicating a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, and determining the model is further based at least in part on the second disparity value. The depth values ​​for training the model may be obtained from an autofocus system, a range imaging system, or other system. For example, in the operations of the method, the first depth value includes an autofocus depth corresponding to a first input image frame captured by the first image sensor. As another example, the method may include determining the first depth value based on range imaging. In some aspects, determining the first depth value includes determining the first depth value based on a time-of-flight (ToF) measurement. In some aspects, determining the first depth value includes determining the first depth value based on a light detection and ranging (LIDAR) measurement.

[0014] In some aspects, determining the corrected image may include determining a transformation matrix for warping the input image frame to the field of view of the other of the first image sensor or the second image sensor, such as may be generated by computer vision processing. Determining the transformation matrix may include determining whether a confidence level associated with the transformation matrix is ​​below a threshold level, the confidence level used to determine whether to use the actual disparity value determined by the computer vision processing or the predicted disparity value based on the model. For example, determining the corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on a determination that the confidence level of the transformation matrix is ​​below a threshold level. The decision regarding the actual disparity value or the predicted disparity value may also or alternatively be based on other image characteristics. For example, the method may include determining that an image characteristic of the input image frame is below a threshold level, and determining the corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on a determination that the image characteristic is below a threshold level. In some aspects, the image characteristic may be a luminance level, such that the method includes a step of determining that the luminance of the input image frame is below a threshold level, and the step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the luminance is below the threshold level.

[0015] The corrected image frame may be output from the processing and stored or displayed to a user. For example, the method may include determining a video sequence including a first image frame from a first image sensor, a second image frame from a second image sensor, and the corrected image frame, where the corrected image frame appears in the video sequence between the first image frame and the second image frame. In some aspects, further processing may be performed during the determination of the corrected image frame prior to the output of the frame. For example, the step of determining the corrected image frame is further based on the image frame of the other of the first image sensor or the second image sensor.

[0016] The method may further include selectively including the data in generating the disparity model. For example, the method may include storing a plurality of disparity values, where the model is based on the plurality of disparity values, and / or replacing a previous value of the plurality of disparity values ​​with the first disparity value based on at least one of some of the plurality of disparity values ​​or a time associated with the previous value.

[0017] In an additional aspect of the present disclosure, an apparatus is disclosed that includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform any of the methods or techniques described herein. For example, the at least one processor may be configured to perform steps including receiving a first disparity value indicating a difference in field of view between a first image sensor and a second image sensor, receiving a first depth value corresponding to the first disparity value, determining a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, the model being based at least in part on the first disparity value and the first depth, receiving an input image frame from one of the first image sensor or the second image sensor, receiving an input image depth corresponding to the input image frame, determining a predicted disparity value corresponding to the input image depth based at least in part on the model, and / or determining a corrected image frame based at least in part on the input image frame and the predicted disparity value. The corrected image frame may be determined as warped to match the field of view of the other of the first image sensor or the second image sensor based on the predicted disparity value. The predicted disparity value may include values ​​of two or more axes, such that the first disparity value indicates a difference in field of view along a first axis, and the method further includes receiving a second disparity value indicating a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, and the determining the model is further based at least in part on the second disparity value. The depth values ​​for training the model may be obtained from an autofocus system, a range imaging system, or other system. For example, in the operations of the method, the first depth value includes an autofocus depth corresponding to a first input image frame captured by the first image sensor. As another example, the method may include determining the first depth value based on range imaging. In some aspects, determining the first depth value includes determining the first depth value based on a time-of-flight (ToF) measurement.In some aspects, determining the first depth value includes determining the first depth value based on light detection and ranging (LIDAR) measurements.

[0018] The at least one processor may include one or more image signal processors and / or one or more processors that include specific functions for camera control and / or processing. The at least one processor may also or alternatively include an application processor. The methods and techniques described herein may be performed entirely by the image signal processor or application processor, or various operations may be split between the image signal processor and the application processor, and in some aspects across additional processors.

[0019] The device may include at least two image sensors, including a first image sensor and a second image sensor, the first image sensor having a larger field of view (FOV) than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor and the second image sensor may be a telephoto image sensor. In another example, the first sensor is configured to acquire images through a first lens having a first optical axis, and the second sensor is configured to acquire images through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification and the second lens may have a second magnification different from the first magnification. This configuration may be performed with a lens cluster on the mobile device, such as when multiple image sensors and associated lenses are located at offset locations on the front or back of the mobile device. Additional image sensors with larger, smaller, or the same field of view may be included. The image correction techniques described herein may be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0020] In an additional aspect of the present disclosure, a device configured for image capture is disclosed. The apparatus includes first and second means for capturing light data (e.g., image frames) representing a scene, such as an image sensor (including a charge-coupled device (CCD), a Bayer filter sensor, an infrared (IR) detector, an ultraviolet (UV) detector, a complementary metal oxide semiconductor (CMOS) sensor). The apparatus may further include one or more means for accumulating and / or focusing light rays onto one or more image sensors (including a simple lens, a compound lens, a spherical lens, and an aspherical lens). The apparatus may also include one or more means for capturing the depth of the scene captured by the means for capturing light data (including a time-of-flight (ToF) system, a light detection and ranging (LIDAR) system, a structured light system, an autofocus system). The device may be configured to perform one or more operations including receiving a first disparity value indicating a difference in field of view between a first image sensor and a second image sensor, receiving a first depth value corresponding to the first disparity value, determining a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, the model being based at least in part on the first disparity value and the first depth, receiving an input image frame from one of the first image sensor or the second image sensor, receiving an input image depth corresponding to the input image frame, determining a predicted disparity value corresponding to the input image depth based at least in part on the model, and / or determining a corrected image frame based at least in part on the input image frame and the predicted disparity value. The corrected image frame may be determined as warped to match the field of view of the other of the first image sensor or the second image sensor based on the predicted disparity value.The predicted disparity values ​​may include values ​​for two or more axes, such that a first disparity value indicates a difference in field of view along a first axis, and the method further includes receiving a second disparity value indicating a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, and determining the model is further based at least in part on the second disparity value. The depth values ​​for training the model may be obtained from an autofocus system, a range imaging system, or other system. For example, in the operations of the method, the first depth value includes an autofocus depth corresponding to a first input image frame captured by the first image sensor. As another example, the method may include determining the first depth value based on range imaging. In some aspects, determining the first depth value includes determining the first depth value based on a time-of-flight (ToF) measurement. In some aspects, determining the first depth value includes determining the first depth value based on a light detection and ranging (LIDAR) measurement.

[0021] In an additional aspect of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations including those described in the methods and techniques described herein. For example, the operations may include receiving a first disparity value indicating a difference in field of view between a first image sensor and a second image sensor, receiving a first depth value corresponding to the first disparity value, determining a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, the model being based at least in part on the first disparity value and the first depth, receiving an input image frame from one of the first image sensor or the second image sensor, receiving an input image depth corresponding to the input image frame, determining a predicted disparity value corresponding to the input image depth based at least in part on the model, and / or determining a corrected image frame based at least in part on the input image frame and the predicted disparity value. The corrected image frame may be determined as warped to match the field of view of the other of the first image sensor or the second image sensor based on the predicted disparity value. The predicted disparity values ​​may include values ​​for two or more axes, such that a first disparity value indicates a difference in field of view along a first axis, and the method further includes receiving a second disparity value indicating a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, and determining the model is further based at least in part on the second disparity value. The depth values ​​for training the model may be obtained from an autofocus system, a range imaging system, or other system. For example, in the operations of the method, the first depth value includes an autofocus depth corresponding to a first input image frame captured by the first image sensor. As another example, the method may include determining the first depth value based on range imaging. In some aspects, determining the first depth value includes determining the first depth value based on a time-of-flight (ToF) measurement. In some aspects, determining the first depth value includes determining the first depth value based on a light detection and ranging (LIDAR) measurement.

[0022] Other aspects, features, and implementations will be apparent to those skilled in the art upon reviewing the following description of certain exemplary aspects in conjunction with the accompanying figures. Although features may be described in conjunction with certain aspects and figures below, various aspects may include one or more of the advantageous features described herein. In other words, although one or more aspects may be described as having certain advantageous features, one or more of such features may also be used in accordance with various aspects. Similarly, although exemplary aspects may be described below as device aspects, system aspects, or method aspects, the exemplary aspects may be implemented in various devices, systems, and methods.

[0023] The method may be embodied in a computer readable medium as computer program code including instructions that cause a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device that includes a first network adapter configured to transmit data over a first network connection of a plurality of network connections, and a processor coupled to the first network adapter and a memory.

[0024] The foregoing has outlined rather broadly some of the features and technical advantages of embodiments of the present invention in order that the detailed description that follows may be better understood. Additional features and advantages of the invention that form the subject of the claims of the invention will be described below. Those skilled in the art will appreciate that the conception and specific embodiments disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same or similar purposes. Those skilled in the art will also realize that such equivalent constructions do not depart from the spirit and scope of the invention as set forth in the appended claims. Additional features will be better understood from the following description when considered in conjunction with the accompanying figures. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended to limit the invention.

[0025] A further understanding of the nature and advantages of the present disclosure may be realized by reference to the following drawings. In the accompanying drawings, similar components or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes the similar components. When only a first reference label is used herein, the description is applicable to any of the similar components having the same first reference label, regardless of the second reference label. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 is a block diagram of a computing device configured to perform one or more of the example techniques described in this disclosure. [Diagram 2] 1 is a block diagram illustrating modeling of image sensor disparity in a multi-sensor device, in accordance with one or more aspects of the present disclosure. [Diagram 3] 1 is a block diagram illustrating warping of an image frame using predicted disparity, in accordance with one or more aspects of the present disclosure. [Figure 4] 1 is a graph illustrating a generated model for image sensor disparity, according to one or more aspects of the present disclosure. [Diagram 5] 1 is a flowchart illustrating a method for correcting an image frame using predicted disparity data, in accordance with one or more aspects of the present disclosure. [Figure 6] 1 is a flowchart illustrating a method for selectively using predicted disparity data in compensating an image frame, according to one or more aspects. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Like reference numbers and designations in the various drawings indicate like elements.

[0028] The detailed description set forth below with reference to the accompanying drawings is for the purpose of illustrating various configurations and is not intended to limit the scope of the present disclosure. Rather, the detailed description includes specific details for the purpose of providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in all cases and that in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0029] The present disclosure provides systems, apparatus, methods, and computer readable media that support image processing of captured image frames for photography and video. Particular implementations of the subject matter described in this disclosure may be implemented to realize potential advantages or benefits, such as improved image quality by reducing artifacts in a sequence of captured image frames during a change from one image sensor to another or other change that affects the field of view of the captured image frames in the sequence of image frames. The systems, apparatus, methods, and computer readable media may be embedded within an image capture device, such as a mobile phone, tablet computing device, laptop computing device, other computing device, or digital camera.

[0030] An exemplary device for capturing image frames using multiple image sensors, such as a smartphone, may include a configuration of two, three, four, or more cameras on the back (e.g., opposite the user display) or front (e.g., on the same side as the user display) of the device. A device with multiple image sensors includes one or more image signal processors (ISPs), computer vision processors (CVPs), or other suitable circuitry for processing images captured by the image sensors. The one or more image signal processors may provide processed image frames to a memory and / or a processor (such as an application processor, an image front end (IFE), an image processing engine (IPE), or other suitable processing circuitry) for further processing, such as for encoding, storage, transmission, or other manipulation.

[0031] As used herein, an image sensor may refer to the image sensor itself, as well as any other suitable components coupled to the image sensor. For example, an image sensor may also refer to other components of a camera, including a shutter, a buffer, or other readout circuitry. An image sensor may further refer to an analog front end, or other circuitry for converting an analog signal to a digital representation of a frame. Thus, the term "image sensor" herein may refer to any suitable components for capturing an image frame and reading out the image frame to an image signal processor.

[0032] In the following description, numerous specific details are set forth, such as examples of specific components, circuits, and processes, to provide a thorough understanding of the present disclosure. The term "coupled" as used herein means directly connected to or connected through one or more intervening components or circuits. Also, in the following description, for purposes of explanation, specific names are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that these specific details may not be required to practice the teachings disclosed herein. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the teachings of the present disclosure. Some portions of the following detailed description are presented in terms of procedures, logic blocks, processes, and other symbolic representations of operations on data bits within a computer memory. In this disclosure, a procedure, logic block, process, etc., is conceived to be a self-consistent sequence of steps or instructions that leads to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0033] It should be borne in mind, however, that all of these and similar terms should be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise indicated, and as will be apparent from the following description, it should be appreciated that throughout this application, descriptions utilizing terms such as "accessing," "receiving," "sending," "using," "selecting," "determining," "normalizing," "multiplying," "averaging," "monitoring," "comparing," "applying," "updating," "measuring," "deriving," "solving," "generating," and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities in the computer system's registers and memory and convert that data to other data similarly represented as physical quantities in the computer system's registers, memory, or other such information storage, transmission, or display device.

[0034] In the figures, a single block may be described as performing one or more functions. The one or more functions performed by the block may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various example components, blocks, modules, circuits, and steps are described below generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Also, the example device may include components other than those shown, including well-known components such as a processor, memory, etc.

[0035] Aspects of the present disclosure are applicable to any suitable electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or "frames"). Additionally, aspects of the present disclosure may be implemented in devices having or coupled to image sensors of the same or different capabilities and characteristics (such as resolution, shutter speed, sensor type, etc.). Additionally, aspects of the present disclosure may be implemented in devices for processing image frames, such as processing devices that may retrieve stored images for processing, including processing devices present in cloud computing systems, regardless of whether the device includes or is coupled to an image sensor.

[0036] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (such as a smartphone, a camera controller, a processing system, etc.). As used herein, a device may be any electronic device having one or more parts that may implement at least some parts of the present disclosure. The following description and examples use the term "device" to describe various aspects of the present disclosure, but the term "device" is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus may include a device or a portion of a device for performing the described operations.

[0037] FIG. 1 illustrates a block diagram of an exemplary device 100 for performing image capture from one or more image sensors. The device 100 may include or be otherwise coupled to an image signal processor 112 for processing image frames from multiple image sensors, such as a first image sensor 101, a second image sensor 102, and a depth sensor 140. The depth sensor 140 may include one or more range imaging devices, such as a time-of-flight (ToF) sensor, a light ranging and detection (LiDAR) system, an infrared (IR) ranging system, a structured light imaging system, and / or a radio frequency (RF) ranging system. In some embodiments, multiple image signal processors may be used instead of a single image signal processor 112, such that each image sensor 101 and 102 is coupled to a different image signal processor. In some embodiments, the additional image sensors may share an image signal processor with one of the two ISPs coupled to the sensors 101 and 102, or additional image signal processors may be coupled to each of the additional image sensors. In some aspects, each of the multiple ISPs may perform operations in parallel on image frames acquired from a respective image sensor, such as those described herein with reference to FIGS. 2-6. In some aspects, one of the multiple ISPs may be a master device and the others may be slave devices, with the master device performing cooperative operations and / or operations performed on data acquired from two or more of the image sensors, such as operations to determine disparity between two image frames acquired from the two image sensors. A computer vision processor (CVP) may be coupled to each of the image sensors to perform computer vision operations on the output of the image sensors, such as feature matching to determine a difference in field of view between a first image sensor and a second image sensor, and provide the difference and / or other feature matching information to the ISP 112 or processor 104, which uses the difference and / or other feature matching information as a disparity value.Feature matching may include operations such as determining a set of feature points in image frames acquired from an image sensor, defining a region around each feature point, and optionally normalizing the area around the feature points to improve accuracy, computing local descriptors of the normalized region, and matching the descriptors between image frames captured by different image sensors. In some aspects, the feature matching algorithm may include one or more of the following: scale-invariant feature transform (SIFT), random sample consensus (RANSAC), binary robust independent elementary features (BRIEF), speeded-up robust features (SURF), and / or oriented FAST and rotated brief (ORB) algorithms.

[0038] In some aspects, the device 100 also includes or is coupled to a processor 104 and a memory 106 that stores instructions 108. The device 100 may also include or be coupled to a display 114 and several input / output (I / O) components 116, such as a touch screen interface and / or physical buttons. The device 100 may further include or be coupled to a power source 118 for the device 100, such as a battery or a component for coupling the device 100 to an energy source. The device 100 may also include or be coupled to additional features or components not shown. In one example, a wireless interface, which may include several transceivers and a baseband processor, may be included for wireless communication devices. In another example, one or more other sensors (such as a global positioning system (GPS) receiver) may be included in or coupled to the device. In a further example, an analog front end for converting analog image frame data to digital image frame data may be coupled between the image sensors 101 and 102 and the image signal processor 112.

[0039] The device may include or be coupled to a sensor hub 150 for interfacing with sensors to receive data regarding the movement of device 100, data regarding the environment surrounding device 100, and / or other non-camera sensor data. Such non-camera sensors may be integrated into and / or coupled to device 100. One exemplary non-camera sensor is a gyroscope, which is a device configured to measure rotation, orientation, and / or angular velocity. Another exemplary non-camera sensor is an accelerometer, which is a device configured to measure acceleration, which may be used to determine speed and distance traveled by appropriately integrating the measured acceleration.

[0040] Image signal processor 112 may receive image data, such as that used to form image frames, from a local bus connection to image sensors 101, 102, or by other connections, such as a wire interface to an external image sensor or a wireless interface to a distant image sensor. In some embodiments, device 100 may include a first camera with a first image sensor 101 and a corresponding first lens 131, and a second camera with a second image sensor 102 and a corresponding second lens 132. Each of lenses 131 and 132 may have an associated autofocus (AF) system 133 and 134, respectively, that adjusts lenses 131 and 132 to focus on a particular focal plane at a fixed scene depth from sensors 101 and 102. AF systems 133 and 134 may be assisted by depth sensor 140 and / or may provide depth information to other components of device 100, such as ISP 112, through metadata associated with image frames captured by sensors 101 and 102. In some embodiments, device 100 may include an interface for receiving image data from image sensors 101 and 102 located remotely from device 100. Device 100 may perform image processing on image data from a combination of image sensors located within device 100 or remotely from device 100.

[0041] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. For example, the first image sensor 101 and the second image sensor 102 may be included in one multi-camera configuration (such as a dual camera configuration, a triple camera configuration, etc. for a smartphone or other suitable device) or in separate single or multiple camera configurations. The image sensors 101 and 102 may also include or be coupled to one or more lenses 131 and 132 for collecting light, one or more apertures for receiving light, one or more shutters for blocking light when outside an exposure window, one or more color filter arrays (CFAs) for filtering light outside a certain frequency range, one or more analog front ends for converting analog measurements into digital information, or other suitable components for imaging. For example, the first image sensor 101 may be coupled to the first lens 131, and the second image sensor 102 may be coupled to the second lens 132. The first lens 131 and the second lens 132 may have different fields of view, such as when the first lens 131 is an ultra-wide angle (UW) lens and the second lens 132 is a wide angle (W) lens. The device 100 may also include or be coupled to a flash, a depth sensor, a GPS, or other suitable components for imaging or supporting imaging applications.

[0042] The multiple image sensors may include a combination of ultra-wide (high field of view (FOV)), wide, telephoto, and super telephoto (low FOV) sensors. That is, each image sensor may be configured through hardware configuration and / or software settings to capture different but overlapping fields of view. In one configuration, the image sensors are configured with different lenses having different magnifications resulting in different fields of view. The sensors may be configured such that a UW sensor has a larger FOV than a W sensor, which has a larger FOV than a T sensor, which has a larger FOV than a UT sensor. For example, a sensor configured for a wide FOV may capture a field of view ranging from 64 to 84 degrees, a sensor configured for an ultra side FOV may capture a field of view ranging from 100 to 140 degrees, a sensor configured for a telephoto FOV may capture a field of view ranging from 10 to 30 degrees, and a sensor configured for a super telephoto FOV may capture a field of view ranging from 1 to 8 degrees. Some aspects of the present disclosure include processing the captured image frames, such as by adjusting the spatial alignment of one or more of the captured image frames, while the device transitions from capturing an image of the scene using a first of the multiple image sensors to capturing an image of the scene using a second of the multiple image sensors. Aspects of the present disclosure may be used to capture image frames using multiple image sensors of an image capture device. For example, camera motion data used for digital image stabilization (DIS) may be acquired using a combination of image frames from the multiple image sensors.

[0043] The image signal processor 112 processes image frames captured by the image sensors 101 and 102. Although FIG. 1 illustrates the device 100 as including two image sensors 101 and 102 coupled to the image signal processor 112, any number of image sensors may be coupled to the image signal processor 112. Additionally, any number of additional image sensors or image signal processors may be present for the device 100. In some embodiments, the image signal processor 112 may execute instructions from a memory, such as instructions 108 from the memory 106, instructions stored in a separate memory coupled to or included in the image signal processor 112, or instructions provided by the processor 104. Additionally or alternatively, the image signal processor 112 may execute software and / or include specific hardware (such as one or more integrated circuits (ICs)) to perform one or more operations described in this disclosure.

[0044] In some implementations, the memory 106 may include a non-transient or non-transitory computer-readable medium that stores computer-executable instructions 108 for performing all or a portion of one or more operations described in this disclosure. In some implementations, the instructions 108 include a camera application (or other suitable application) to be executed by the device 100 to generate images or videos. The instructions 108 may also include other applications or programs executed by the device 100, such as an operating system or a specific application other than for image or video generation. Execution of the camera application, such as by the processor 104, may cause the device 100 to generate images using the image sensors 101 and 102 and the image signal processor 112. The memory 106 may also be accessed by the image signal processor 112 to store processed frames or by the processor 104 to obtain processed frames. In some embodiments, the device 100 does not include the memory 106. For example, device 100 may be a circuit that includes image signal processor 112 and the memory may be external to device 100. Device 100 may be coupled to the memory and configured to access the memory to write output frames for display or long-term storage.

[0045] In some embodiments, the processor 104 may include one or more general-purpose processors capable of executing scripts or instructions of one or more software programs, such as the instructions 108 stored in the memory 106. For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in the memory 106. In executing the camera application, the processor 104 may be configured to instruct the image signal processor 112 to perform one or more operations related to the image sensor 101 or 102. For example, the camera application may receive a capture command, whereupon a video including a sequence of image frames is captured and processed by one or more methods described herein for reducing artifacts by warping some of the image frames. Execution of the instructions 108 by the processor 104 outside of the camera application may also cause the device 100 to perform any number of functions or operations. In some embodiments, the processor 104 may include ICs or other hardware in addition to the ability to execute software to cause the device 100 to perform some functions or operations, such as those described herein. In some other embodiments, the device 100 does not include the processor 104, such as when all of the described functionality is configured in the image signal processor 112.

[0046] In some embodiments, at least one of the image signal processor 112 or the processor 104 may execute instructions to perform various operations described herein. For example, execution of the instructions may instruct the image signal processor 112 to begin or end capturing an image frame or a sequence of image frames. As another example, execution of the instructions may instruct the image signal processor 112 to change from capturing a first image of a scene captured using the first image sensor 101 to a second image of the scene captured using the second image sensor 102. As a further example, execution of the instructions may instruct the image signal processor 112 to change a zoom level from a first zoom level to a second zoom level, which may result in switching from a first image sensor having a first field of view to a second image sensor having a second field of view. As yet another example, execution of the instructions may instruct the image signal processor 112 to switch from a first image sensor for capturing image frames to a second image sensor for capturing image frames during recording of a sequence of image frames.

[0047] In some embodiments, display 114 may include one or more suitable displays or screens that allow for user interaction and / or presenting items to a user, such as previews of image frames being captured by image sensors 101 and 102. In some embodiments, display 114 is a touch-sensitive display. I / O components 116 may be or include any suitable mechanism, interface, or device for receiving input (such as commands) from a user and for providing output to a user. For example, I / O components 116 may include (without being limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a compressible bezel, one or more buttons (such as a power button), sliders, switches, etc.

[0048] Although shown coupled together via the processor 104, the processor 104, memory 106, image signal processor 112, display 114, and I / O components 116 may be coupled together in various other configurations, such as via one or more local buses not shown for simplicity. Although the image signal processor 112 is shown as separate from the processor 104, the image signal processor 112 may be a core of the processor 104, an application processor unit (APU) included in a system on chip (SoC) or otherwise included in the processor 104. Although the device 100 is referred to in examples herein for carrying out aspects of the present disclosure, some device components may not be shown in FIG. 1 to avoid obscuring aspects of the present disclosure. In addition, other components, multiple components, or combinations of components may be included in a suitable device for carrying out aspects of the present disclosure. Thus, the present disclosure is not limited to a particular device or configuration of components, including the device 100.

[0049] A multi-sensor device such as apparatus 100 can switch from one image sensor to another, such as between sensors 101 and 102, based on a user request or certain criteria being met. For example, a user can indicate a request to switch from a wide-angle (W) lens corresponding to sensor 101 to a telephoto (T) lens corresponding to sensor 102 by activating a portrait mode in a camera application accessing sensors 101 and 102. As another example, a user can change the zoom level in the camera application, which causes the image signal processor 112 to switch from sensor 101 to sensor 102 based on the characteristics of the lenses corresponding to sensors 101 and 102. As a further example, changing scene characteristics such as light levels can cause the image signal processor 112 to switch from sensor 101 to sensor 102 to achieve better light sensitivity.

[0050] In any of these exemplary (or other) switching between image sensors, artifacts may appear in the sequence of image frames at or around the change between the sensors due to differences in the field of view of the sensors. The shortcomings noted herein are merely representative and are included to highlight problems that the inventors have identified with existing devices and have sought to ameliorate. Aspects of the devices described below may address some or all of the shortcomings, as well as others known in the art. Aspects of the improved devices described herein offer benefits other than those described above and may be used in applications other than those described above.

[0051] In one embodiment of the device 100, the image frames captured from the sensors 101 and 102 may be corrected to improve field of view alignment between the sensors 101 and 102. For example, an input image frame from one sensor may be geometrically warped based on the disparity, i.e., difference, between the image frames captured by the sensors 101 and 102. The disparity between the image frames may arise due to different positions of the image sensors 101 and 102 and imperfect alignment between the sensors 101 and 102. The disparity between the image frames may be determined and used as a basis to determine the amount and manner of geometric warping to be applied during generation of the corrected image frames. The actual disparity between the image frames may be determined by feature matching between a first image frame captured from a first image sensor and a second image frame captured from a second image sensor. The feature matching may be performed using computer vision (CV).

[0052] The feature matching process may fail due to sensor failure, scene conditions, scene changes, and / or other conditions. When feature matching fails, disparity values ​​for determining geometric warping are unavailable or inaccurate, which may result in poor performance of geometric warping. As a result, the sequence of image frames may have noticeable artifacts, or in the worst case, image frames may be missing from the sequence. A model may be built from known disparity values, such as those determined during a calibration routine of the device and / or preloaded on the device during a previous image capture on the device. The model may be used to provide predicted disparity values ​​to complement or replace disparity values ​​determined from feature matching or other disparity determination techniques. Model generation is described in one embodiment of FIG. 2.

[0053] FIG. 2 is a block diagram illustrating modeling of image sensor disparity in a multi-sensor device, according to one or more aspects of the present disclosure. The system 200 illustrates generation of a model 212 in a disparity predictor 210. A computer vision (CV) processor (CVP) 202 receives a first image frame and a second image frame capturing a scene from different perspectives. The image frames may be received at the CV processor 202 from an image sensor, such as through the image signal processor 112, or may be retrieved from a memory. The CV processor 202 may analyze the images to identify image features and correlate matching image features from the first image frame to the second image frame. Based on the matching features, the CV processor 202 may determine a shift of the features from the first image frame to the second image frame. The shift may be expressed as one or more disparity values, such as a first disparity value along a first axis and a second disparity value along a second axis. In some aspects, the first and second axes may be orthogonal, such as when the first and second axes are x-axis and y-axis. The determined disparity values, such as dx and dy values, may be provided to the disparity predictor 210 by the CV processor 202. The predictor 210 may also receive a scene depth corresponding to the scene reflected in the first and second image frames from which the dx and dy values ​​were determined. The scene depth may reflect a distance from the image sensor to the object of interest, as measured by a depth sensor. The scene depth may alternatively or additionally be received as an autofocus distance corresponding to the first or second image frames from an autofocus sensor. The predictor 210 may apply the scene depth and disparity values ​​to train the model 212. The operations of the disparity predictor 210 may be performed by an image signal processor (ISP), such as the ISP 112. The operations of the disparity predictor 210 may alternatively be performed by a processor, such as a digital signal processor (DSP) or processor 104.

[0054] The model 212 may be a nonlinear regression model, a linear regression model, or a machine learning algorithm. The machine learning model may include, in some embodiments, a logistic regression technique, a linear discriminant analysis, a linear regression analysis, an artificial neural network, a machine learning classifier algorithm, or a classification / regression tree, as described herein. In some aspects, the machine learning may include one or more artificial neural networks, which may include an interconnected group of artificial neurons (e.g., neuron models) to model the relationship between parameters such as disparity and scene depth. In some aspects, the machine learning may include one or more convolutional neural networks, which is a type of feed-forward artificial neural network. A convolutional neural network may include a collection of neurons, each having a receptive field, that collectively tile the input space. In some aspects, the machine learning may include one or more deep learning architectures, such as deep belief networks and deep convolutional networks, which are layered neural network architectures, where the output of a first layer of neurons becomes the input to a second layer of neurons, the output of the second layer of neurons becomes the input to a third layer of neurons, and so on. The deep neural network may be trained to recognize a hierarchy of features. In various aspects, the machine learning system may employ some kind of naive Bayes predictive modeling analysis, learning vector quantization, or implementation of a boosting algorithm, such as Adaboost or stochastic gradient boosting system, to iteratively update weights to train a machine learning classifier to determine the relationship between influence attributes, such as scene depth and disparity, and / or the degree to which such influence attributes affect the outcome of such a system or disparity value.

[0055] After training, the disparity predictor 210 may use the model 212 to predict disparity values ​​for pairs of image frames captured from the image device 100. In some aspects, the display predictor 210 may be executed by logic circuitry of the device 100 to obtain training of the model 212 based on image frames captured by the image sensors 101 and 102. In some aspects, the display predictor 210 may be executed by a remote cloud-based computing system. When the model 212 is generated in the cloud, it may be exported and used on one or more image capture devices, such as the device 100. In some aspects, the model 212 may be trained in the cloud from data captured from other image capture devices to generate a generic model that is loaded on an individual device 100 and then revised by a predictor on the individual device based on the unique characteristics of each device. For example, dropping a device may result in changes in disparity between image sensors that are specific to a particular device and may be accounted for in the local revision of the model 212.

[0056] Although the CV processor 202 is shown as generating disparity values ​​for training the model 212, other techniques may be used to generate training data. Regardless of the manner in which the model 212 is trained, the disparity predictor 210 may use the estimator 214 to generate predicted disparity values ​​using the model 212. An exemplary generation and use of predicted disparity values ​​is illustrated in FIG. 3. FIG. 3 is a block diagram illustrating warping of an image frame using predicted disparity, according to one or more aspects of the present disclosure. The system 300 uses the disparity predictor 210 to generate one or more predicted disparity values. The estimator 214 of the predictor 210 may receive a scene depth value for which a predicted disparity value is desired. For example, an autofocus depth corresponding to the image frame being processed may be input to the estimator 214. The estimator 214 may access the model 212 to obtain predicted disparity values ​​dx and dy for the input scene depth.

[0057] The predicted disparity values ​​may be input to a geometric warp module 310 to correct the input image frame. The warp module 310 receives the input image frame and the predicted disparity values. The warp module 310 then determines a corrected image frame 330 based at least in part on the predicted disparity values ​​and the input image frame. The corrected image frame 330 may be further processed before, after, or in parallel with the processing of the warp module 310. Other processing of the corrected image frame may include computational photography, high dynamic range (HDR), multi-frame noise reduction (MFNR), low-light enhancement, and super-resolution processing. Such computational photography may include processing in which the determination of the corrected image frame includes fusing the input image frame (after applying warping with a transformation matrix to match another image sensor) with an image frame from another image sensor and / or additional image frames from the same image sensor to improve the appearance of the corrected image frame. Examples of such computational photography include obtaining improved dynamic range from fusing one or more additional image frames from one or more other image sensors, obtaining reduced noise from fusing one or more additional image frames from one or more other image sensors, and / or obtaining higher resolution from fusing one or more additional image frames from one or more other image sensors. The corrected image frame 330 may be geometrically warped to improve alignment of the field of view of the input image frame to other image frames in the sequence 340 of image frames captured at or around a switch from a first image sensor to a second image sensor having a different field of view than the first image sensor. The corrected image frame 330 may appear in a video sequence of image frames between a first image frame captured from a first image sensor and a second image frame captured from a second image sensor.

[0058] One exemplary model 212 for generation of predicted disparity values ​​by the estimator 214 is illustrated in FIG. 4. FIG. 4 is a graph illustrating a generated model for image sensor disparity, according to one or more aspects of the present disclosure. The graph 400 includes a first line 402 reflecting a first predicted disparity value and a second line 404 reflecting a second predicted disparity value. The first and second lines 402 and 404 may reflect disparity values ​​along different axes, such as the dx and dy disparity values ​​described above. The first line 402 may be generated from data 412 collected during training of the model 212, and the second line 404 may be generated from data 414 collected during training of the model 212.

[0059] The individual data points for the data 412 and 414 may be stored in memory in the predictor 210, in the memory 106, in the cloud, or in other storage locations. In some aspects, the data 412 and 414 may be used to update the training of the model 212, such as when new data is received. In some aspects, the model is generated on-demand from the stored data 412 and 414 when a predicted disparity value is required. In some aspects, the amount of data stored is a predetermined number of sets of disparity values ​​with corresponding scene depths. As new data is received, older sets may be discarded based on an algorithm such as first-in-first-out (FIFO) or an assignment priority associated with the set of values. In some aspects, the amount of data stored is limited to a certain period of time such that sets of data that exceed a certain age are removed from the stored data. In some aspects, a combination of time, number, and other criteria may be used to determine the age of the stored data 412 and 414 to generate the model 212. For example, data from an accelerometer may be used to determine that a drop incident has occurred that may cause the model 212 to no longer be accurate, and as a result, data received prior to the drop incident should be removed or de-weighted in generating the model 212. As another example, a certain number of disparity samples for different depth ranges may be stored, such as 10 disparity values ​​for a depth of 0-10 centimeters, another 10 disparity values ​​for a depth of 10-20 centimeters, another 10 disparity values ​​for a depth of 20-50 centimeters, and 20 values ​​for a depth of 50 centimeters to infinity. The model 212 may be updated by a least squares method each time new data is added, old data is aged out, and / or old data is replaced with new data. The model 212 reflecting the lines 402 and 404 may be represented by values, such as equations and / or matrices and / or coefficients corresponding to parameters of the machine learning algorithm, and their values ​​stored in the predictor 210.

[0060] One embodiment of a method for generating and applying aspects of the model 212 described herein for correcting an image frame is described with reference to FIG. 5. FIG. 5 is a flow chart illustrating a method for correcting an image frame using predicted disparity data according to one or more aspects of the present disclosure. The method 500 begins at block 502 with receiving a first disparity value indicative of a difference in field of view between a first image sensor and a second image sensor at a corresponding first depth. At block 504, a second disparity value indicative of a difference in field of view between the first image sensor and a second image sensor at a corresponding second depth is received. A model may be constructed from just two scene depths with corresponding disparity values.

[0061] At block 506, a model for disparity between the first image sensor and the second image sensor is determined based on the first disparity value, the first depth, the second disparity value, and the second depth. The first disparity value and the first depth may form a first training set, and the second disparity and the second depth may form a second training set. The training set may include disparity values ​​having the same depth value. The model may have a machine learning algorithm that learns the relationship between disparity and scene depth from the training set. The model may alternatively be a programmed equation with coefficients, constants, and / or other parameters that are adjusted when the training set is received to reflect the relationship between disparity and scene depth. Thus, the model may be constructed with only two training sets, such as when the model is a linear equation. The model may be reconstructed for higher order equations when additional training sets are available.

[0062] At block 508, the model is applied to generate predicted disparity values ​​for the input image frames. For example, depths corresponding to the input image frames may be input into the model to generate one or more predicted disparity values ​​that predict disparity values ​​for a difference in field of view between the first image sensor and the second image sensor. In some aspects, the input image frames may be image frames captured by one of the first image sensor or the second image sensor. In some aspects, the input image frames may be image frames captured by an image sensor corresponding to one of the first or second image sensors, such as a similar sensor on a device different from the device from which the training data for the model was acquired. Predicted disparity values ​​for the image frames captured from one of the first and / or second image sensors may be generated and compared to actual disparity measured from the image sensor to provide feedback to adjust the model. The predicted disparity values ​​may be used to correct image frames captured from one or both of the first and second image sensors. In one exemplary application of the predicted disparity values, the predictions may be used to geometrically warp one or both of the first and second image sensors.

[0063] At block 510, the input image frame is warped based on the predicted disparity value of block 508. Warping may involve transforming a grid of the input image frame to align with the field of view of the other image sensor during image sensor switching. In one aspect, the predicted disparity value may be used to generate an artificial computer vision processing matrix to replace the output of the CV processor. In one aspect, the predicted disparity value may be applied on a crop window to achieve non-center cropping to reduce shifts that need to occur in the computer vision processing stage. When the warped image frame is viewed as a video in a sequence of image frames, the warped image frame may reduce the appearance of jumps, skips, or other artifacts around the image sensor switching. The image warping may continue across two or more image frames of a video sequence to smooth the transition from one image sensor to another, such as by scaling the geometric warping with a scaling factor that decreases toward unity for each subsequent frame. When the scaling factor reaches a unity value, such that the warped image frame based on the current image sensor aligns with the field of view of another image sensor, the image sensor can switch to the other image sensor. After the switch, processing of image frames from the other image sensor may continue, and the preview or recorded video continues. Although warping is described as a method for determining a corrected image frame from an input image frame, other processing of the input image frame may be performed in addition to or as an alternative to warping. Some of the processing may be based on predicted disparity values, and some of the processing may be based on other values.

[0064] In some aspects, the predicted disparity values ​​may be used to correct all image frames output from the image signal processor. In some aspects, the predicted disparity values ​​may be selectively used during operation of the image capture device to determine output frames to be corrected. For example, the use of the predicted disparity values ​​may be turned on for a certain period of time and turned off for another period of time. The "off" period of time may be used to train the model, and then the predicted disparity values ​​may be turned back on. For example, a user may enter a calibration mode, in which the device requests the user to enter a well-lit room and take a series of photos of an object at several depths by walking toward the object to obtain a training set. As another example, criteria for input image frames may be evaluated, and the evaluation of those criteria used to determine whether to use the predicted disparity values ​​generated from the model for any particular input image frame. One aspect for selectively applying the predicted disparity values ​​in determining corrected image frames is described with reference to FIG. 6.

[0065] FIG. 6 is a flow chart illustrating a method of selectively using predicted disparity data in correcting an image frame according to one or more aspects. The method 600 includes receiving a first input image frame and a second input image frame at block 602. At block 604, image features in one or both of the first and second input image frames are analyzed. The analysis at block 604 may include feature matching between the first input image frame and the second input image frame. The difference between the locations of the matched features from the first and second image frames may be used as the actual disparity in warping the image frame. However, the image matching may fail, resulting in a subsequent failure of the image warping, which causes artifacts when the image frame is viewed as a video in a sequence of image frames. The analysis at block 604 may be considered to determine whether to use the actual disparity and / or the predicted disparity in processing the input image frame.

[0066] In block 606, criteria are evaluated to determine whether the predicted disparity should be used to determine the corrected image frame. The criteria may include whether the feature matching was successful or unsuccessful. The criteria may also or alternatively include whether the confidence level of the feature matching is above a threshold level. Other criteria based on the characteristics of the image frames may also be used. For example, the average luminance of the first and / or second input image frames may be determined and compared to a threshold to determine whether the image is too dark for high confidence in the image matching. As another example, the focus position and / or confidence of the lens capturing the first and / or second image frames may be determined and compared to a threshold to determine whether the image is too out of focus for high confidence in the image matching. In some aspects, the feature matching of block 604 may generate a transformation matrix for the input image frame to warp the input image frame to the field of view of the other of the first image sensor or the second image sensor. The confidence level of the transformation matrix may be compared to a threshold level to evaluate whether to use the predicted disparity.

[0067] If the criteria are met at block 606, the method 600 continues to block 608, which determines a predicted disparity at a scene depth corresponding to one or both of the first and second input image frames. Then, at block 610, the first input image frame may be warped to the second input image frame, or the second input image frame may be warped to the first input image frame, based on the predicted disparity of block 608.

[0068] If the criteria for using the predicted disparity at block 606 is not met, the method 600 continues to block 612, where a disparity is determined using feature matching for the first and second input image frames. Then, at block 614, the first input image frame may be warped to the second input image frame, or the second input image frame may be warped to the first input image frame, based on the determined disparity at block 612.

[0069] In one or more aspects, a technique for supporting image capture with reduced artifacts, such as jumps and skips of objects in a scene recorded in a video sequence, resulting in higher quality photos and videos generated by the device. Improved video quality may be provided by obtaining more accurate disparity values ​​while warping an image frame captured by one image sensor to align it with the field of view of another image sensor to facilitate switching from one image sensor to another. In addition, the apparatus may perform one or more aspects or operate according to one or more aspects as described below. In some implementations, the apparatus includes a wireless device, such as a UE. In some implementations, the apparatus may include at least one processor and a memory coupled to the processor. The processor may be configured to perform operations described herein with respect to the apparatus. In some other implementations, the apparatus may include a non-transitory computer-readable medium having program code recorded thereon, the program code being executable by a computer to cause a computer to perform operations described herein with respect to the apparatus. In some implementations, the apparatus may include one or more means configured to perform operations described herein. In some implementations, a method of wireless communication may include one or more of the operations described herein with respect to an apparatus.

[0070] In one or more aspects, a technique for supporting image capture with reduced artifacts such as jumps and skips of objects in a scene recorded in a video sequence resulting in higher quality photos and videos generated by the device. Improved video quality may be provided by obtaining more accurate disparity values ​​while warping image frames captured by one image sensor to align with the field of view of another image sensor to facilitate switching from one image sensor to another. In one or more aspects, supporting image capture may include an apparatus configured to receive a first disparity value indicating a difference in field of view between a first image sensor and a second image sensor. The apparatus is further configured to receive a first depth value corresponding to the first disparity value. The apparatus is further configured to determine a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, the model being based at least in part on the first disparity value and the first depth. Additionally, the apparatus may perform or operate according to one or more aspects as described below. In some implementations, the apparatus includes a wireless device such as a UE. In some implementations, the apparatus may include at least one processor and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the apparatus. In some other implementations, the apparatus may include a non-transitory computer-readable medium having program code recorded thereon, the program code being executable by a computer to cause the computer to perform the operations described herein with respect to the apparatus. In some implementations, the apparatus may include one or more means configured to perform the operations described herein. In some implementations, a method of wireless communication may include one or more operations described herein with respect to the apparatus and may include transmitting the corrected image frame to another mobile device, to a base station, or to a server, either through the base station or directly to a server or other mobile device.

[0071] In a first aspect, an apparatus is configured to receive an input image frame from one of a first image sensor or a second image sensor.

[0072] In a second aspect, in combination with the first aspect, an apparatus is configured to receive an input image depth corresponding to an input image frame.

[0073] In a third aspect, in combination with the second aspect, the apparatus is configured to determine a predicted disparity value corresponding to an input image depth based at least in part on the model.

[0074] In a fourth aspect, in combination with the third aspect, the apparatus is configured to determine the corrected image frame based at least in part on the input image frame and the predicted disparity value.

[0075] In a fifth aspect, in combination with one or more of the first to fourth aspects, determining the corrected image frame includes determining the corrected image frame as warped to match the field of view of the other of the first image sensor or the second image sensor based on the predicted disparity value.

[0076] In a sixth aspect, in combination with one or more of the first to fifth aspects, determining the corrected image frame includes determining a transformation matrix for warping the input image frame to a field of view of the other of the first image sensor or the second image sensor.

[0077] In a seventh aspect, in combination with the sixth aspect, determining the transformation matrix includes determining the transformation matrix using computer vision processing (CVP).

[0078] In an eighth aspect, in combination with the sixth aspect, the apparatus is configured to determine whether a confidence level associated with the transformation matrix is ​​below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on a determination that the confidence level of the transformation matrix is ​​below the threshold level.

[0079] In a ninth aspect, in combination with one or more of the first to eighth aspects, the apparatus is configured to determine that an image characteristic of the input image frame is below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on a determination that the image characteristic is below the threshold level.

[0080] In a tenth aspect, in combination with one or more of the first to ninth aspects, the apparatus is configured to determine that a luminance of the input image frame is below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the luminance is below the threshold level.

[0081] In an eleventh aspect, in combination with one or more of the first to tenth aspects, the apparatus is configured to determine a video sequence including a first image frame from a first image sensor, a second image frame from a second image sensor, and a correction image frame, the correction image frame appearing in the video sequence between the first image frame and the second image frame.

[0082] In a twelfth aspect, in combination with one or more of the first to eleventh aspects, the step of determining the corrected image frame is further based on an image frame of the other of the first image sensor or the second image sensor.

[0083] In a thirteenth aspect, alone or in combination with one or more of the first to twelfth aspects, the first parallax value indicates a difference in field of view along a first axis.

[0084] In a fourteenth aspect, in combination with the thirteenth aspect, the method is configured to receive a second disparity value indicative of a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, and the determining of the model is further based at least in part on the second disparity value.

[0085] In a fifteenth aspect, alone or in combination with one or more of the first to fourteenth aspects, determining the model includes storing a plurality of disparity values. The model is based on the plurality of disparity values.

[0086] In a sixteenth aspect, in combination with the fifteenth aspect, a previous value of the plurality of disparity values ​​is replaced with the first disparity value based on at least one of some of the plurality of disparity values ​​or a time associated with the previous value.

[0087] In a seventeenth aspect, alone or in combination with one or more of the first to sixteenth aspects, the first depth value includes an autofocus depth corresponding to a first input image frame captured by the first image sensor.

[0088] In an eighteenth aspect, alone or in combination with one or more of the first to seventeenth aspects, the apparatus is configured to determine a first depth value based on range imaging.

[0089] In a nineteenth aspect, in combination with the eighteenth aspect, determining the first depth value includes determining the first depth value based on a time-of-flight (ToF) measurement.

[0090] In a twentieth aspect, in combination with one or more of the eighteenth to nineteenth aspects, determining the first depth value includes determining the first depth value based on light detection and ranging (LIDAR) measurements.

[0091] In one or more aspects, the techniques for supporting image capture may include additional aspects, such as any single aspect or any combination of aspects described below or with respect to one or more other processes or devices described elsewhere herein. In one or more aspects, supporting image capture may include an apparatus configured for a processor. The apparatus is further configured for a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the device to perform operations including receiving a first disparity value indicating a difference in field of view between a first image sensor and a second image sensor. The apparatus is further configured for receiving a first depth value corresponding to the first disparity value. The apparatus is further configured for determining a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, the model being based at least in part on the first disparity value and the first depth. Additionally, the apparatus may perform one or more aspects or operate according to one or more aspects as described below. In some implementations, the apparatus includes a wireless device, such as a base station. In some implementations, the apparatus may include at least one processor and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the apparatus. In some other implementations, the apparatus may include a non-transitory computer-readable medium having program code recorded thereon, the program code being executable by a computer to cause the computer to perform the operations described herein with respect to the apparatus. In some implementations, the apparatus may include one or more means configured to perform the operations described herein. In some implementations, the method of wireless communication may include one or more operations described herein with respect to the apparatus.

[0092] In a twenty-first aspect, the instructions cause the device to perform operations including receiving an input image frame from one of the first image sensor or the second image sensor.

[0093] In a twenty-second aspect, in combination with the twenty-first aspect, an apparatus is configured to receive an input image depth corresponding to an input image frame.

[0094] In a twenty-third aspect, in combination with the twenty-second aspect, the apparatus is configured to determine a predicted disparity value corresponding to an input image depth based at least in part on the model.

[0095] In a twenty-fourth aspect, in combination with the twenty-third aspect, the apparatus is configured to determine the corrected image frame based at least in part on the input image frame and the predicted disparity value.

[0096] In a 25th aspect, in combination with one or more of the 21st to 24th aspects, determining the corrected image frame includes determining a corrected image frame warped to match the field of view of the other of the first image sensor or the second image sensor based on the predicted disparity value.

[0097] In a 26th aspect, in combination with one or more of the 21st to 25th aspects, determining the corrected image frame includes determining a corrected image frame warped to match the field of view of the other of the first image sensor or the second image sensor based on the predicted disparity value.

[0098] In a 27th aspect, in combination with one or more of the 21st to 26th aspects, determining the corrected image frame includes determining a transformation matrix for warping the input image frame to a field of view of the other of the first image sensor or the second image sensor.

[0099] In a twenty-eighth aspect, in combination with the twenty-seventh aspect, determining the transformation matrix includes determining the transformation matrix using computer vision processing (CVP).

[0100] In a twenty-ninth aspect, in combination with one or more of the twenty-seventh to twenty-eighth aspects, the instructions cause the device to perform operations further including determining whether a confidence level associated with the transformation matrix is ​​below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on a determination that the confidence level of the transformation matrix is ​​below the threshold level.

[0101] In a 30th aspect, in combination with one or more of the 21st to 29th aspects, the instructions cause the device to perform operations further including determining that an image characteristic of the input image frame is below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on a determination that the image characteristic is below the threshold level.

[0102] In a thirty-first aspect, in combination with one or more of the twenty-first to thirtieth aspects, the instructions cause the device to perform operations further including determining that a luminance of the input image frame is below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on a determination that the luminance is below the threshold level.

[0103] In a thirty-second aspect, in combination with one or more of the twenty-first to thirty-first aspects, the instructions cause the device to perform operations further including determining a video sequence including a first image frame from a first image sensor, a second image frame from a second image sensor, and a correction image frame, the correction image frame appearing in the video sequence between the first image frame and the second image frame.

[0104] In a 33rd aspect, in combination with one or more of the 21st to 32nd aspects, the step of determining the corrected image frame is further based on an image frame of the other of the first image sensor or the second image sensor.

[0105] In a thirty-fourth aspect, alone or in combination with one or more of the twenty-first to thirty-fourth aspects, the first parallax value indicates a difference in field of view along a first axis.

[0106] In a thirty-fifth aspect, in combination with the thirty-third aspect, the instructions cause the device to perform an operation further including receiving a second disparity value indicative of a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, and determining the model is further based at least in part on the second disparity value.

[0107] In a thirty-sixth aspect, alone or in combination with one or more of the twenty-first to thirty-fifth aspects, determining the model includes storing a plurality of disparity values. The model is based on the plurality of disparity values.

[0108] In a thirty-seventh aspect, in combination with the thirty-sixth aspect, a previous value of the plurality of disparity values ​​is replaced with the first disparity value based on at least one of some of the plurality of disparity values ​​or a time associated with the previous value.

[0109] In a 38th aspect, alone or in combination with one or more of the 21st to 37th aspects, the first depth value includes an autofocus depth corresponding to a first input image frame captured by the first image sensor.

[0110] In a thirty-ninth aspect, alone or in combination with one or more of the twenty-first to thirty-eighth aspects, the instructions cause the device to perform an operation further including determining a first depth value based on range imaging.

[0111] In a fortieth aspect, in combination with the thirty-ninth aspect, determining the first depth value includes determining the first depth value based on a time-of-flight (ToF) measurement.

[0112] In a forty-first aspect, in combination with the fortieth aspect, determining the first depth value includes determining the first depth value based on light detection and ranging (LIDAR) measurements.

[0113] In one or more aspects, techniques for supporting image capture may include additional aspects, such as any single aspect or any combination of aspects described below or with respect to one or more other processes or devices described elsewhere herein. In one or more aspects, supporting image capture may include an apparatus configured to cause a device to perform an operation including receiving a first disparity value indicating a difference in field of view between a first image sensor and a second image sensor. The apparatus is further configured to perform an operation including receiving a first depth value corresponding to the first disparity value. The apparatus is further configured to perform an operation including determining a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, the model being based at least in part on the first disparity value and the first depth. In addition, the apparatus may perform one or more aspects or operate according to one or more aspects as described below. In some implementations, the apparatus includes a wireless device, such as a base station. In some implementations, the apparatus may include at least one processor and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the apparatus. In some other implementations, the apparatus may include a non-transitory computer-readable medium having program code recorded thereon, the program code being executable by a computer to cause the computer to perform the operations described herein with respect to the apparatus. In some implementations, the apparatus may include one or more means configured to perform the operations described herein. In some implementations, a method of wireless communication may include one or more operations described herein with respect to the apparatus.

[0114] In a forty-second aspect, the instructions, when executed by a processor of the device, cause the device to perform operations further including receiving an input image frame from one of the first image sensor or the second image sensor.

[0115] A forty-third aspect, in combination with the forty-second aspect, includes receiving an input image depth corresponding to an input image frame.

[0116] A forty-fourth aspect, in combination with the forty-third aspect, determines a predicted disparity value corresponding to the input image depth based at least in part on the model.

[0117] The forty-fifth aspect, in combination with the forty-fourth aspect, determines a corrected image frame based at least in part on the input image frame and the predicted disparity value.

[0118] In a 46th aspect, in combination with one or more of the 42nd to 45th aspects, determining the corrected image frame includes determining a corrected image frame warped to match the field of view of the other of the first image sensor or the second image sensor based on the predicted disparity value.

[0119] In a 47th aspect, in combination with one or more of the 42nd to 46th aspects, determining the corrected image frame includes determining a transformation matrix for warping the input image frame to a field of view of the other of the first image sensor or the second image sensor.

[0120] In a forty-eighth aspect, in combination with the forty-seventh aspect, determining the transformation matrix includes determining the transformation matrix using computer vision processing (CVP).

[0121] In a forty-ninth aspect, in combination with one or more of the forty-seventh to forty-eighth aspects, the instructions, when executed by a processor of a device, cause the device to perform operations further including determining whether a confidence level associated with the transformation matrix is ​​below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on a determination that the confidence level of the transformation matrix is ​​below the threshold level.

[0122] In a 50th aspect, in combination with one or more of the 42nd to 49th aspects, the instructions, when executed by a processor of the device, cause the device to perform operations further including determining that an image characteristic of the input image frame is below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on a determination that the image characteristic is below the threshold level.

[0123] In a 51st aspect, in combination with one or more of the 42nd aspect to the 50th aspect, the instructions, when executed by a processor of the device, cause the device to perform operations further including determining that a luminance of the input image frame is below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on a determination that the luminance is below the threshold level.

[0124] In a 52nd aspect, in combination with one or more of the 42nd to 51st aspects, the instructions, when executed by a processor of the device, cause the device to perform operations further including determining a video sequence including a first image frame from a first image sensor, a second image frame from a second image sensor, and a correction image frame, the correction image frame appearing in the video sequence between the first image frame and the second image frame.

[0125] In a 53rd aspect, in combination with one or more of aspects 42 to 52, the step of determining the corrected image frame is further based on an image frame of the other of the first image sensor or the second image sensor.

[0126] In a fifty-fourth aspect, alone or in combination with one or more of the forty-second to fifty-third aspects, the first parallax value indicates a difference in field of view along a first axis.

[0127] In a fifty-fifth aspect, in combination with the fifty-fourth aspect, the instructions, when executed by a processor of the device, cause the device to perform operations further including receiving a second disparity value indicative of a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, and determining the model is further based at least in part on the second disparity value.

[0128] In a fifty-sixth aspect, alone or in combination with one or more of the forty-second to fifty-fifth aspects, determining the model includes storing a plurality of disparity values. The model is based on the plurality of disparity values.

[0129] In a 57th aspect, in combination with the 56th aspect, a previous value of the plurality of disparity values ​​is replaced with the first disparity value based on at least one of some of the plurality of disparity values ​​or a time associated with the previous value.

[0130] In a 58th aspect, alone or in combination with one or more of aspects 42 to 57, the first depth value includes an autofocus depth corresponding to a first input image frame captured by the first image sensor.

[0131] In a 59th aspect, alone or in combination with one or more of aspects 42 to 58, the instructions, when executed by a processor of the device, cause the device to perform operations further including determining a first depth value based on range imaging.

[0132] In a sixtieth aspect, in combination with the fifty-ninth aspect, determining the first depth value includes determining the first depth value based on a time-of-flight (ToF) measurement.

[0133] In a 61st aspect, in combination with one or more of the 59th to 60th aspects, determining the first depth value includes determining the first depth value based on light detection and ranging (LIDAR) measurements.

[0134] In one or more aspects, techniques for supporting image capture may include additional aspects, such as any single aspect or any combination of aspects described below or with respect to one or more other processes or devices described elsewhere herein. In one or more aspects, supporting image capture may include an apparatus configured with a first image sensor configured with a first field of view, a second image sensor configured with a second field of view that at least partially overlaps the first field of view, a processor coupled to the first image sensor and coupled to the second image sensor, and a memory coupled to the processor. The apparatus is further configured to perform steps including receiving a first disparity value indicative of a difference in field of view between the first image sensor and the second image sensor. The apparatus is further configured to receive a first depth value corresponding to the first disparity value. The apparatus is further configured to determine a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, the model being based at least in part on the first disparity value and the first depth. In addition, the apparatus may perform one or more aspects or operate according to one or more aspects as described below. In some implementations, the apparatus includes a wireless device, such as a base station. In some implementations, the apparatus may include at least one processor and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the apparatus. In some other implementations, the apparatus may include a non-transitory computer-readable medium having program code recorded thereon, the program code being executable by a computer to cause the computer to perform the operations described herein with respect to the apparatus. In some implementations, the apparatus may include one or more means configured to perform the operations described herein. In some implementations, a method of wireless communication may include one or more operations described herein with respect to the apparatus.

[0135] In a sixty-second aspect, the processor is further configured to perform steps including receiving an input image frame from one of the first image sensor or the second image sensor.

[0136] In a sixty-third aspect, in combination with the sixty-second aspect, an input image depth corresponding to an input image frame is received.

[0137] The 64th aspect, in combination with the 63rd aspect, determines a predicted disparity value corresponding to the input image depth based at least in part on the model.

[0138] The sixty-fifth aspect, in combination with the sixty-fourth aspect, determines a corrected image frame based at least in part on the input image frame and the predicted disparity value.

[0139] In a 66th aspect, in combination with one or more of the 62nd to 65th aspects, determining the corrected image frame includes determining a corrected image frame warped to match the field of view of the other of the first image sensor or the second image sensor based on the predicted disparity value.

[0140] In a 67th aspect, in combination with the 66th aspect, determining the corrected image frame includes determining a transformation matrix for the input image frame for warping the input image frame to a field of view of the other of the first image sensor or the second image sensor.

[0141] In a sixty-eighth aspect, in combination with the sixty-seventh aspect, determining the transformation matrix includes determining the transformation matrix using computer vision processing (CVP).

[0142] In a sixty-ninth aspect, in combination with one or more of the sixty-seventh to sixty-eighth aspects, the processor of the fifth aspect is further configured to perform a step including determining whether a confidence level associated with the transformation matrix is ​​below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on a determination that the confidence level of the transformation matrix is ​​below the threshold level.

[0143] In a seventieth aspect, alone or in combination with one or more of the sixty-second to sixty-ninth aspects, the processor is further configured to perform a step including determining that an image characteristic of the input image frame is below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on a determination that the image characteristic is below the threshold level.

[0144] In a seventy-first aspect, alone or in combination with one or more of the sixty-second to seventy-first aspects, the processor of the fifth aspect is further configured to perform a step including determining that a luminance of the input image frame is below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the luminance is below the threshold level.

[0145] In a seventy-second aspect, alone or in combination with one or more of the sixty-second to seventy-first aspects, the processor is further configured to perform a step including determining a video sequence including a first image frame from the first image sensor, a second image frame from the second image sensor, and a correction image frame, the correction image frame appearing in the video sequence between the first image frame and the second image frame.

[0146] In a 73rd aspect, alone or in combination with one or more of aspects 62 to 72, the step of determining the corrected image frame is further based on an image frame of the other of the first image sensor or the second image sensor.

[0147] In a seventy-fourth aspect, alone or in combination with one or more of the sixty-second to seventy-third aspects, the first parallax value indicates a difference in field of view along a first axis.

[0148] In a seventy-fifth aspect, in combination with the seventy-fourth aspect, the processor is configured to perform a step further including receiving a second disparity value indicative of a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, and the step of determining the model is further based at least in part on the second disparity value.

[0149] In a seventy-sixth aspect, alone or in combination with one or more of the sixty-second to seventy-fifth aspects, determining the model includes storing a plurality of disparity values. The model is based on the plurality of disparity values.

[0150] In a 77th aspect, in combination with the 76th aspect, a previous value of the plurality of disparity values ​​is replaced with the first disparity value based on at least one of some of the plurality of disparity values ​​or a time associated with the previous value.

[0151] In a seventy-eighth aspect, alone or in combination with one or more of the sixty-second to seventy-seventh aspects, an apparatus is configured for a depth sensor. A processor is coupled to the depth sensor and configured to receive a first depth value from the depth sensor.

[0152] In a seventy-ninth aspect, in combination with the seventy-eighth aspect, the depth sensor includes a range imaging system.

[0153] In an eightieth aspect, in combination with the seventy-ninth aspect, the depth sensor includes a time-of-flight (ToF) system. Determining the first depth value includes determining the first depth value based on ToF measurements from the ToF system.

[0154] In an eighty-first aspect, in combination with one or more of the seventy-ninth to eightyth aspects, the depth sensor includes a light detection and ranging (LIDAR) system. Determining the first depth value includes determining the first depth value based on LIDAR measurements from the LIDAR system.

[0155] In an eighty-second aspect, alone or in combination with one or more of the sixty-second to eighty-first aspects, an apparatus is configured for a computer vision processor (CVP) coupled to a processor, the CVP configured to perform operations including receiving a first image frame from a first image sensor, receiving a second image frame from a second image sensor, and feature matching between the first and second image frames to determine a difference in field of view between the first and second image sensors. The processor is configured to receive the difference in field of view between the first and second image sensors as a first disparity value from the CVP.

[0156] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0157] The components, functional blocks, and modules described herein with respect to Figures 1, 2, and 3 include, among other examples, processors, electronic devices, hardware devices, electronic components, logical circuits, memories, software code, firmware code, or any combination thereof. In addition, features described herein may be implemented via dedicated processor circuitry, via executable instructions, or a combination thereof.

[0158] Those skilled in the art will further appreciate that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the present disclosure herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Those skilled in the art will also readily appreciate that the order or combination of components, methods, or interactions described herein are merely examples, and that the components, methods, or interactions of various aspects of the present disclosure may be combined or performed in ways other than those shown and described herein.

[0159] The various example logic, logic blocks, modules, circuits, and algorithmic processes described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. Interchangeability of hardware and software has been described generally in terms of functionality and illustrated in the various example components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the particular application and design constraints imposed on the overall system.

[0160] The hardware and data processing devices used to implement the various example logic, logic blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed using general purpose single-chip or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some implementations, particular processes and methods may be performed by circuitry specific to a given function.

[0161] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, or any combination thereof, including the structures disclosed herein and their structural equivalents. Implementations of the subject matter described herein may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or for controlling the operation of a data processing apparatus.

[0162] If implemented in software, the functions may be stored on or transmitted over a computer-readable medium as one or more instructions or code. The processes of the methods or algorithms disclosed herein may be implemented in a processor-executable software module that may reside on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that may enable a computer program to be transferred from one place to another. A storage medium may be any available medium that may be accessed by a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Also, any connection may be properly referred to as a computer-readable medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically and discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media. In addition, operations of a method or algorithm may reside on machine-readable and computer-readable media, which may be incorporated into a computer program product as one or any combination or set of code and instructions.

[0163] Various modifications of the implementations described in this disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to several other implementations without departing from the spirit or scope of this disclosure. Thus, the claims are not intended to be limited to the implementations shown herein, but should be accorded the widest scope consistent with this disclosure, the principles and novel features disclosed herein.

[0164] In addition, those skilled in the art will readily appreciate that the terms "upper" and "lower" may be used to simplify illustration of the figures, indicate relative positions that correspond to the orientation of the figure on a suitably oriented page, and may not reflect the proper orientation of any implemented device.

[0165] Some features described herein in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations separately or in any suitable subcombination. Furthermore, although features may be described above as working in some combinations and may even be initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0166] Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order depicted or in sequential order, or that all depicted operations be performed, to achieve desirable results. Additionally, the figures may generally depict another exemplary process in the form of a flow diagram. However, other operations not depicted may be incorporated into the generally depicted exemplary process. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the depicted operations. In some circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged within multiple software products. Additionally, some other implementations fall within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results.

[0167] As used herein, including in the claims, the term "or," when used in a list of two or more items, means that any one of the listed items may be employed alone, or any combination of two or more of the listed items may be employed. For example, if a composition is described as including components A, B, or C, the composition may include only A, only B, only C, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B, and C. Also, as used herein, including in the claims, "or" used in a list of items ending with "at least one of" indicates a disjunctive list, such as, for example, a list of "at least one of A, B, or C" means any of A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any of these in any combination thereof. The term "substantially" is defined as the majority of, but not necessarily the entirety of, what is specified (and includes what is specified, e.g., substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by those skilled in the art. In any disclosed implementation, the term "substantially" may be replaced with "within [a percentage] of" what is specified, where the percentage includes 0.1, 1, 5, or 10 percent.

[0168] The preceding description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications of the disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. [Explanation of symbols]

[0169] 100 devices 101 first image sensor, image sensor, sensor 102 second image sensor, image sensor, sensor 104 processors 106 Memory 108 Computer executable instructions, instructions 112 Image Signal Processor, ISP 114 Display 116 Input / Output (I / O) Components, I / O Components 118 Power supply 131 First Lens, Lens 132 Second Lens, Lens 133 Autofocus (AF) system, AF system 134 Autofocus (AF) system, AF system 140 Depth Sensor 150 Sensor Hub 200 Systems 202 Computer Vision (CV) Processor (CVP), CV Processor 210 Disparity predictor, predictor 212 Model 214 Estimator 300 Systems 310 Geometric Warp Module, Warp Module 330 Corrected Image Frames 340 Sequence 400 graphs 402 First Line, Line 404 Second Line, Line 412 Data 414 Data 500 ways 600 ways

Claims

1. receiving a first disparity value indicative of a difference in field of view between a first image sensor and a second image sensor; receiving a first depth value corresponding to the first disparity value; determining a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, the model being based at least in part on the first disparity value and a first depth; receiving an input image from one of the first image sensor or the second image sensor; receiving a second depth value corresponding to the input image; determining a predicted disparity value based on the model and the second depth value; determining a transformation matrix for warping the input image to a field of view of the other of the first image sensor or the second image sensor; determining that a confidence level associated with the transformation matrix is ​​below a threshold level; determining a corrected image based on the input image and the predicted disparity value; The method includes:

2. 2. The method of claim 1 , wherein determining the corrected image comprises warping the input image received from one of the first image sensor or the second image sensor to match a field of view of the other of the first image sensor or the second image sensor.

3. The method of claim 1 , wherein determining the transformation matrix comprises determining the transformation matrix using computer vision processing (CVP).

4. determining that an image characteristic of the input image is below a second threshold level; Further comprising: determining the corrected image is performed based on a determination that the image characteristic is below the second threshold level. The method of claim 1.

5. determining when the luminance of the input image is below a third threshold level; Further comprising: determining the corrected image is performed based on a determination that the luminance is below the third threshold level. The method of claim 1.

6. determining a video sequence including a first image from the first image sensor, a second image from the second image sensor, and the compensated image, wherein the compensated image appears in the video sequence between the first image and the second image. The method of claim 1, further comprising:

7. The method of claim 1 , wherein determining the corrected image is further based on an image of the other of the first image sensor or the second image sensor.

8. the first disparity value indicating a difference in field of view along a first axis; The method further comprising: receiving a second disparity value indicative of a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, wherein determining the model is further based at least in part on the second disparity value.

3. The method of claim 2, further comprising:

9. determining the model comprises: storing a plurality of disparity values, the model being based on the plurality of disparity values; replacing the previous value of the plurality of disparity values ​​with the first disparity value based on at least one of a value of the plurality of disparity values ​​or a time associated with the previous value; 2. The method of claim 1, comprising:

10. The method of claim 1 , wherein the first depth value comprises an autofocus depth corresponding to the input image.

11. determining the first depth value based on range imaging; Optionally, the step of determining the first depth value comprises determining the first depth value based on a Time of Flight (ToF) measurement or a Light Detection and Ranging (LIDAR) measurement.

12. A device, comprising: one or more processors; and a memory coupled to the one or more processors that stores instructions that, when executed by the one or more processors, cause the device to: receiving a first disparity value indicative of a difference in field of view between a first image sensor and a second image sensor; receiving a first depth value corresponding to the first disparity value; determining a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, the model being based at least in part on the first disparity value and a first depth; receiving an input image from one of the first image sensor or the second image sensor; receiving a second depth value corresponding to the input image; determining a predicted disparity value based on the model and the second depth value; determining a transformation matrix for warping the input image to a field of view of the other of the first image sensor or the second image sensor; determining that a confidence level associated with the transformation matrix is ​​below a threshold level; determining a corrected image based on the input image and the predicted disparity value; A device that causes an action to be performed, including 13. The device of claim 12, further configured to perform a method according to any one of claims 2 to 11.

14. A method for detecting an image of a subject, comprising: detecting a first image sensor configured with a first field of view; and detecting a second image sensor configured with a second field of view that at least partially overlaps with the first field of view. a depth sensor, the one or more processors coupled to the depth sensor and configured to receive the first depth value from the depth sensor; a computer vision processor (CVP) coupled to the one or more processors, receiving a first image from the first image sensor; receiving a second image from the second image sensor; matching features between the first image and the second image to determine a difference in field of view between the first image sensor and the second image sensor; and a CVP configured to perform operations including:

13. The device of claim 12, wherein the one or more processors are configured to receive the difference in field of view between the first image sensor and the second image sensor as the first disparity value from the CVP.

15. A non-transitory computer readable medium storing instructions that, when executed by a processor of a device, cause the device to perform operations including the method of any one of claims 1 to 11.

Images