Transformation matrix learning for multi-sensor image capture devices

Abstract

Artifacts in a sequence of image frames may be reduced or eliminated by modification to align an input image frame with another image frame in the sequence, such as by geometrically warping to generate a corrected image frame having a field of view aligned with another frame in the sequence of frames having image frames. The warping may be performed based on a model generated from data relating to a multi-sensor device. Disparity between image frames may be modeled based on image capture from a first and second image sensor for a scene at various depths. The model may be used to predict disparity values ​​of captured images, and those predicted disparity values ​​may be used to reduce artifacts resulting from image sensor switching. The predicted disparity values ​​may be used in image conditions that result in erroneous actual disparity values.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Patent Application No. 17 / 201,660, filed on March 15, 2021, entitled "TRANSFORM MATRIX LEARNING FOR MULTI - SENSOR IMAGE CAPTURE DEVICES", which is hereby incorporated by reference in its entirety.

[0002] Aspects of the present disclosure generally relate to image processing. Some features of the present disclosure can enable and provide improvements in the processing of images obtained from multi - sensor image capture devices.

Background Art

[0003] Devices that can capture one or more digital images, whether still - image photographs or sequences of images for video, can be incorporated into a wide variety of devices. By way of example, image capture devices can include stand - alone digital cameras or digital video camcorders, mobile phones, cellular or satellite radiotelephones, personal digital assistants (PDAs), panel or tablet computers, camera - equipped wireless communication device handsets such as game devices, webcams, computer devices such as video surveillance cameras, or other devices having digital imaging or video capabilities.

[0004] Some image capture devices include multiple image sensors that capture image data through one or more lenses; such devices are called multi-sensor image capture devices or multi-sensor devices. Multiple image sensors may consist of different lenses to provide multiple fields of view of a scene and / or different zoom levels of the scene. Exemplary lens types include wide-angle lenses, ultra-wide-angle lenses, telephoto lenses, telescope lenses, periscope zoom lenses, fisheye lenses, macro lenses, prime lenses, or various combinations thereof. For example, a dual-camera configuration may include both a wide-angle lens and a telephoto lens.

[0005] However, since users are generally not interested in multiple images of a scene, but rather in capturing and displaying a single image, the use of multiple image sensors increases the complexity of image processing in the device. Therefore, multiple frames captured from multiple image sensors may be processed to produce a single image for the user. Nevertheless, due to the different physical characteristics of the image sensors, the frames acquired from each image sensor may flow together in a way that the transition from one image sensor to another is perceptible to the human eye. For example, changing from a telephoto lens to a wide-angle lens while capturing a series of image frames for display as a video may result in the appearance of scene shift in the video. Artifacts such as scene shift in the output of a multi-sensor image capture device are undesirable. [Overview of the project] [Means for solving the problem]

[0006] Artifacts in video frames result from a break in the continuity of image frames output from a multi-sensor device, caused by switching from one image sensor to another image sensor with a different field of view. Artifacts can be reduced or eliminated by modifying the input image frame to align it with the other image frame. For example, the input frame may be geometrically warped by stretching, shrinking, cropping, and / or otherwise modifying it to generate a corrected image frame with a field of view aligned with another frame in a sequence of frames containing the image frame. Image frame modification may be performed based on a model generated from data about the multi-sensor device.

[0007] Warping can be based on a shift between the field of view from a first image sensor to a second image sensor, called parallax. Parallax can be modeled based on image captures from the first and second image sensors for scenes at various depths. The model may be used to predict parallax values ​​for captured images, and these predicted parallax values ​​are used to reduce artifacts resulting from image sensor switching. Predicted parallax values ​​may be used in image conditions that would result in incorrect actual parallax values. For example, if the actual parallax value is determined to be unavailable or incorrect, the predicted parallax value can substitute for the actual parallax value. This substitution can reduce artifacts resulting from failures in image sensor switching and / or other components associated with image capture, such as computer vision processing (CVP). Since predicted parallax values ​​are available under conditions where actual parallax determination may fail, the predicted parallax values ​​determined from the model can facilitate switching image sensors for any zoom ratio and any scene.

[0008] In some embodiments, a single model may be used to model the parallax of all image sensors. In some embodiments, a separate model may be used for each pair of image frames captured from two image sensors. For example, on a device with three image sensors, a first model may be determined corresponding to the first and second image sensors, a second model may be determined for the first and third image sensors, and a third model may be determined for the second and third image sensors. Thus, switching from one image sensor to any of the other may have predictive parallax values ​​available to warp the image frames from one image sensor to match those from the other.

[0009] The following summarizes several aspects of this disclosure in order to provide a basic understanding of the technology described. This summary is not a comprehensive overview of all intended features of this disclosure, nor does it identify the main or important elements of all aspects of this disclosure, nor does it define the scope of any or all aspects of this disclosure. Its sole purpose is to present, in summary form, some concepts of one or more aspects of this disclosure as a prelude to the more detailed explanations that will be presented later.

[0010] Generally, this disclosure describes image processing techniques involving a digital camera having an image sensor and an image signal processor (ISP). To achieve improved image quality with reduced artifacts during image sensor switching, the ISP may warp image frames received from the image sensor using a disparity model. Warping and disparity modeling may, alternatively, be performed on image frames stored separately from the image sensor. Warping and disparity modeling may be performed based on requests from other components in the device, such as by the CPU in response to image processing functions running on the CPU.

[0011] An image signal processor may be configured to control the capture of image frames from one or more image sensors and to process the image frames from one or more image sensors to generate a view of the scene in a corrected image frame. For example, the image signal processor may receive instructions to capture a sequence of image frames in response to the loading of software, such as a camera application, on the CPU. The image signal processor may be configured to generate a single flow of output frames based on each corrected image from the image sensors. The single flow of output frames may include image frames, which contain image data from the image sensors that have been corrected, such as by geometric warping, to align the image frame with other image frames in the output flow (e.g., previous frames captured by different image sensors).

[0012] After the image signal processor generates output frames representing a scene, the view of the scene can be displayed on a device display, stored on a storage device as a sequence of pictures as a picture or video, transmitted over a network, and / or printed on an output medium. For example, an image signal processor may be configured to take input frames of image data (e.g., pixel values) from different image sensors and then generate corresponding output frames of the image data (e.g., preview display frames, still image captures, video frames, etc.). In other examples, the image signal processor may output frames of image data to various output devices and / or camera modules for further processing, such as 3A parameter synchronization, generating video files via output frames, configuring frames for display, or configuring frames for storage. That is, an image signal processor may take incoming frames from one or more image sensors, each coupled to one or more camera lenses, and then generate a flow of output frames, which can be output to various output destinations. In such examples, the image signal processor may be configured to generate a flow of output frames in which artifacts resulting from changes in image sensors may be reduced. In one example, an image signal processor may receive inputs for pinch-zoom actions, changes in zoom level based on gesture detection, or other user inputs to a device containing an image sensor or a user device coupled to a device containing an image sensor. When the image sensor changes in response to user input such as pinch-zoom actions, predictive parallax may be used to warp image frames and reduce artifacts in the sequence of image frames output from the sensor.

[0013] In one aspect of the present disclosure, the method includes the steps of: receiving a first disparity value indicating a difference in the field of view between a first image sensor and a second image sensor; receiving a first depth value corresponding to the first disparity value; determining a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, wherein the model is at least partially based on the first disparity value and the first depth; receiving an input image frame from one of the first image sensor or the second image sensor; receiving an input image depth corresponding to the input image frame; determining a predicted disparity value corresponding to the input image depth based at least partially on the model; and / or determining a corrected image frame based at least partially on the input image frame and the predicted disparity value. The corrected image frame may be determined as warped to match the field of view of the other of the first or second image sensor based on the predicted disparity value. The predicted disparity value may include values ​​for two or more axes such that the first disparity value indicates the difference in the field of view along a first axis, and the method further includes the step of receiving a second disparity value indicating the difference in the field of view between a first image sensor and a second image sensor along a second axis different from the first axis, and determining a model, the step of further determining a model, which is at least partially based on the second disparity value. Depth values ​​for training the model may be obtained from an autofocus system, a range imaging system, or other system. For example, in the operation of the method, the first depth value includes the autofocus depth corresponding to a first input image frame captured by the first image sensor. As another example, the method may include the step of determining the first depth value based on range imaging. In some embodiments, determining the first depth value includes determining the first depth value based on time-of-flight (ToF) measurements. In some embodiments, determining the first depth value includes determining the first depth value based on light detection and ranging (LIDAR) measurements.

[0014] In some embodiments, determining a corrected image may include determining a transformation matrix for warping an input image frame to the field of view of either a first or second image sensor, such that it may be generated by computer vision processing. Determining the transformation matrix may include determining whether the confidence level associated with the transformation matrix is ​​below a threshold level, which is used to determine whether to use actual disparity values ​​determined by computer vision processing or model-based predicted disparity values. For example, the step of determining a corrected image frame based at least in part on the input image frame and predicted disparity values ​​is performed based on the determination that the confidence level of the transformation matrix is ​​below a threshold level. The decision regarding actual or predicted disparity values ​​may, likewise, be based on other image characteristics. For example, the method may include the step of determining that the image characteristics of the input image frame are below a threshold level, and the step of determining a corrected image frame based at least in part on the input image frame and predicted disparity values ​​is performed based on the determination that the image characteristics are below a threshold level. In some embodiments, the image characteristic may be a luminance level, and as a result, the method includes the step of determining that the luminance of an input image frame is below a threshold level, and the step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value, which is performed based on the determination that the luminance is below a threshold level.

[0015] The corrected image frame may be output from the processing and stored or displayed to the user. For example, the method may include the step of determining a video sequence including a first image frame from a first image sensor, a second image frame from a second image sensor, and a corrected image frame, wherein the corrected image frame appears in the video sequence between the first image frame and the second image frame. In some embodiments, further processing may be performed during the determination of the corrected image frame before outputting the frame. For example, the step of determining the corrected image frame may be further based on the image frame of the other of the first or second image sensor.

[0016] The method may further include a step of selectively including data in the generation of a disparity model. For example, the method may include a step of storing a plurality of disparity values, wherein the model is based on the plurality of disparity values, and / or a step of replacing a previous value among the plurality of disparity values ​​with a first disparity value based on at least one of a step and / or a time associated with some of the values ​​or a previous value among the plurality of disparity values.

[0017] In additional aspects of the present disclosure, a device is disclosed that includes at least one processor and memory coupled to at least one processor. The at least one processor is configured to perform any of the methods or techniques described herein. For example, the at least one processor may be configured to perform steps including receiving a first disparity value indicating a difference in the field of view between a first image sensor and a second image sensor; receiving a first depth value corresponding to the first disparity value; and determining a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, wherein the model is determined, at least in part, based on the first disparity value and the first depth; receiving an input image frame from either the first or the second image sensor; receiving an input image depth corresponding to the input image frame; determining a predicted disparity value corresponding to the input image depth, at least in part, based on the model; and / or determining a corrected image frame, at least in part, based on the input image frame and the predicted disparity value. A corrected image frame may be determined based on a predicted disparity value, warped to match the field of view of the first or second image sensor, the other of the two. The predicted disparity value may include values ​​for two or more axes such that the first disparity value indicates the difference in the field of view along a first axis, and the method further includes a step of receiving a second disparity value indicating the difference in the field of view between the first and second image sensors along a second axis different from the first axis, and determining a model, which is further at least partially based on the second disparity value. Depth values ​​for training the model may be obtained from an autofocus system, a range imaging system, or other system. For example, in the operation of the method, the first depth value includes the autofocus depth corresponding to a first input image frame captured by the first image sensor. As another example, the method may include a step of determining the first depth value based on range imaging. In some embodiments, determining the first depth value includes determining the first depth value based on time-of-flight (ToF) measurements.In some embodiments, determining a first depth value includes determining a first depth value based on light detection and ranging (LIDAR) measurements.

[0018] At least one processor may include one or more image signal processors and / or one or more processors, each containing specific functions for camera control and / or processing. At least one processor may also, or alternatively, include an application processor. The methods and techniques described herein may be performed entirely by an image signal processor or an application processor, or various operations may be divided between the image signal processor and the application processor, and in some embodiments, across additional processors.

[0019] The device may include at least two image sensors, including a first image sensor and a second image sensor, wherein the first image sensor has a larger field of view (FOV) than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a telephoto image sensor. In another example, the first sensor is configured to acquire an image through a first lens having a first optical axis, and the second sensor is configured to acquire an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. This configuration can be achieved using a lens cluster on a mobile device, for example, when multiple image sensors and associated lenses are located in offset locations on the front or back of the mobile device. Additional image sensors with a larger field of view, a smaller field of view, or the same field of view may be included. The image correction techniques described herein may be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0020] In additional aspects of this disclosure, a device configured for image capture is disclosed. The device includes first and second means for capturing optical data representing a scene (e.g., image frames), such as image sensors (including charge-coupled devices (CCDs), Bayer filter sensors, infrared (IR) detectors, ultraviolet (UV) detectors, and complementary metal-oxide-semiconductor (CMOS) sensors). The device may further include one or more means for accumulating and / or focusing light rays onto one or more image sensors (including simple lenses, composite lenses, spherical lenses, and aspherical lenses). The device may also include one or more means for capturing the depth of the scene captured by the means for capturing optical data (including time-of-flight (ToF) systems, light detection and ranging (LIDAR) systems, structured light systems, and autofocus systems). The device may be configured to perform one or more operations, including receiving a first disparity value indicating the difference in fields of view between a first image sensor and a second image sensor, receiving a first depth value corresponding to the first disparity value, and determining a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, wherein the model is determined, at least in part, based on the first disparity value and the first depth, receiving an input image frame from one of the first or second image sensors, receiving an input image depth corresponding to the input image frame, determining a predicted disparity value corresponding to the input image depth based at least in part on the model, and / or determining a corrected image frame based at least in part on the input image frame and the predicted disparity value. The corrected image frame may be determined, based on the predicted disparity value, as warped to match the field of view of the other of the first or second image sensor.The predicted disparity value may include values ​​for two or more axes such that the first disparity value indicates the difference in the field of view along a first axis, and the method further includes the step of receiving a second disparity value indicating the difference in the field of view between a first image sensor and a second image sensor along a second axis different from the first axis, and determining a model, the step of further determining a model, which is at least partially based on the second disparity value. Depth values ​​for training the model may be obtained from an autofocus system, a range imaging system, or other system. For example, in the operation of the method, the first depth value includes the autofocus depth corresponding to a first input image frame captured by the first image sensor. As another example, the method may include the step of determining the first depth value based on range imaging. In some embodiments, determining the first depth value includes determining the first depth value based on time-of-flight (ToF) measurements. In some embodiments, determining the first depth value includes determining the first depth value based on light detection and ranging (LIDAR) measurements.

[0021] In additional aspects of the present disclosure, a non-temporary computer-readable medium, when executed by a processor, stores instructions causing the processor to perform operations including those described in the methods and techniques described herein. For example, operations may include receiving a first disparity value indicating the difference in fields of view between a first image sensor and a second image sensor; receiving a first depth value corresponding to the first disparity value; determining a model for disparity between the first image sensor and the second image sensor at a plurality of depth values, wherein the model is determined to be at least partially based on the first disparity value and the first depth; receiving an input image frame from either the first or second image sensor; receiving an input image depth corresponding to the input image frame; determining a predicted disparity value corresponding to the input image depth based at least partially on the model; and / or determining a corrected image frame based at least partially on the input image frame and the predicted disparity value. The corrected image frame may be determined based on the predicted disparity value as warped to match the field of view of the other of the first or second image sensor. The predicted disparity value may include values ​​for two or more axes such that the first disparity value indicates the difference in the field of view along a first axis, and the method further includes the step of receiving a second disparity value indicating the difference in the field of view between a first image sensor and a second image sensor along a second axis different from the first axis, and determining a model, the step of further determining a model, which is at least partially based on the second disparity value. Depth values ​​for training the model may be obtained from an autofocus system, a range imaging system, or other system. For example, in the operation of the method, the first depth value includes the autofocus depth corresponding to a first input image frame captured by the first image sensor. As another example, the method may include the step of determining the first depth value based on range imaging. In some embodiments, determining the first depth value includes determining the first depth value based on time-of-flight (ToF) measurements. In some embodiments, determining the first depth value includes determining the first depth value based on light detection and ranging (LIDAR) measurements.

[0022] Other embodiments, features, and implementations will become apparent to those skilled in the art upon consideration of the following descriptions of specific exemplary embodiments, together with the accompanying figures. Features may be described in relation to some of the embodiments and figures below, but various embodiments may include one or more of the advantageous features described herein. In other words, one or more embodiments may be described as having several advantageous features, but one or more of such features may also be used according to various embodiments. Similarly, exemplary embodiments may be described below as device embodiments, system embodiments, or method embodiments, but exemplary embodiments may be implemented in various devices, systems, and methods.

[0023] The method may be embedded in a computer-readable medium as computer program code containing instructions that cause a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device that includes a first network adapter configured to transmit data over a first network connection among a plurality of network connections, and a processor coupled to the first network adapter and memory.

[0024] The above outlines some of the features and technical advantages of embodiments of the present invention fairly broadly so that the following detailed description may be better understood. Additional features and advantages that form the subject matter of the claims of the present invention are described below. Those skilled in the art will understand that the disclosed concepts and particular embodiments may be readily used as a basis for modifying or designing other structures to accomplish the same or similar objectives. Those skilled in the art will also recognize that such equivalent configurations will not depart from the spirit and scope of the invention as set out in the appended claims. The additional features will be better understood from the following description when considered together with the accompanying drawings. However, it should be clearly understood that each of the drawings is provided for illustrative and explanatory purposes only and does not limit the present invention.

[0025] A further understanding of the nature and advantages of the present disclosure can be realized by referring to the following drawings. In the accompanying drawings, similar components or features may have the same reference labels. Further, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes the similar components. If only the first reference label is used herein, the description is applicable to any of the similar components having the same first reference label regardless of the second reference label.

Brief Description of the Drawings

[0026] [Figure 1] A block diagram of a computing device configured to execute one or more of the exemplary techniques described in the present disclosure. [Figure 2] A block diagram showing the modeling of image sensor parallax in a multi-sensor device according to one or more aspects of the present disclosure. [Figure 3] A block diagram showing the warping of an image frame using predicted parallax according to one or more aspects of the present disclosure. [Figure 4] A graph showing a generated model for image sensor parallax according to one or more aspects of the present disclosure. [Figure 5] A flowchart showing a method for correcting an image frame using predicted parallax data according to one or more aspects of the present disclosure. [Figure 6] A flowchart showing a method for selectively using predicted parallax data in the correction of an image frame according to one or more aspects.

Modes for Carrying Out the Invention

[0027] Similar reference numbers and names in the various drawings indicate similar elements.

[0028] The detailed description below with respect to the attached drawings is intended to illustrate various configurations and does not limit the scope of this disclosure. Rather, the detailed description includes specific details to give a complete understanding of the subject matter of the invention. It will be apparent to those skilled in the art that these specific details are not necessary in all cases, and that in some instances, well-known structures and components are shown in the form of block diagrams for clarity of presentation.

[0029] This disclosure provides systems, apparatus, methods, and computer-readable media for supporting image processing of captured image frames for photographs and videos. Certain implementations of the subject matter described herein may be implemented to realize potential advantages or benefits, such as improved image quality by reducing artifacts in a sequence of captured image frames during changes from one image sensor to another or other changes affecting the field of view of the captured image frames during a sequence of image frames. The systems, apparatus, methods, and computer-readable media may be embedded in image capture devices such as mobile phones, tablet computing devices, laptop computing devices, other computing devices, or digital cameras.

[0030] An exemplary device for capturing image frames using multiple image sensors, such as a smartphone, may include a configuration of two, three, four, or more cameras on the back (e.g., opposite the user display) or front (e.g., the same side as the user display) of the device. A device with multiple image sensors includes one or more image signal processors (ISPs), computer vision processors (CVPs), or other suitable circuitry for processing images captured by the image sensors. One or more image signal processors may provide the processed image frames to memory and / or processors (such as an application processor, image front-end (IFE), image processing engine (IPE), or other suitable processing circuitry) for further processing, such as encoding, storage, transmission, or other operations.

[0031] As used herein, "image sensor" may refer to the image sensor itself and any other suitable components coupled to the image sensor. For example, "image sensor" may also refer to other components of the camera, including a shutter, buffer, or other readout circuitry. "Image sensor" may further refer to an analog front end, or other circuitry for converting analog signals into a digital representation of a frame. Therefore, the term "image sensor" as used herein may refer to any suitable component for capturing image frames and reading image frames to an image signal processor.

[0032] The following description includes numerous specific details, such as examples of specific components, circuits, and processes, to provide a complete understanding of the disclosure. As used herein, the term “combined” means directly connected to or connected through one or more intervening components or circuits. Furthermore, specific names are provided for illustrative purposes to provide a complete understanding of the disclosure. However, it will be apparent to those skilled in the art that these specific details may not be necessary to practice the teachings disclosed herein. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the teachings of the disclosure. Some parts of the following detailed description present procedures, logical blocks, processes, and other symbolic representations of operations on data bits in computer memory. In this disclosure, procedures, logical blocks, processes, etc., are considered to be a self-consistent sequence of steps or instructions that produce a desired result. A step requires the physical manipulation of a physical quantity. These quantities, though not always, take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0033] However, it should be kept in mind that all of these terms and similar terms should be associated with appropriate physical quantities and are merely convenient labels given to those quantities. Unless otherwise specified, as will be evident from the following descriptions, throughout this application, descriptions using terms such as “access,” “receive,” “send,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” “derive,” “solve,” and “generate” refer to actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities in the registers and memory of a computer system and convert that data into other data similarly represented as physical quantities in the registers, memory, or other such information storage, transmission, or display devices of a computer system.

[0034] In the diagrams, a single block may be described as performing one or more functions. The one or more functions performed by that block may be performed in a single component or across multiple components, and / or using hardware, software, or a combination of hardware and software. To clearly demonstrate this hardware-software compatibility, various exemplary components, blocks, modules, circuits, and steps are described below in relation to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A person skilled in the art may implement the described functions in various ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure. Furthermore, the exemplary devices may include components other than those shown, including well-known components such as processors and memory.

[0035] Aspects of this disclosure are applicable to any suitable electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or "frames"). Furthermore, aspects of this disclosure may be implemented in devices having or coupled to image sensors with the same or different capabilities and characteristics (such as resolution, shutter speed, and sensor type). Furthermore, aspects of this disclosure may be implemented in devices for processing image frames, such as processing devices that can retrieve stored images for processing, including processing devices residing in a cloud computing system, regardless of whether the device includes or is coupled to image sensors.

[0036] The terms “device” and “apparatus” are not limited to one or a specific number of physical objects (such as one smartphone, one camera controller, or one processing system). As used herein, a device can be any electronic device having one or more components that can implement at least some parts of this disclosure. The following descriptions and examples use the term “device” to illustrate various aspects of this disclosure, but the term “device” is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus can include a device or part of a device for performing the operation described.

[0037] Figure 1 shows a block diagram of an exemplary device 100 for performing image capture from one or more image sensors. Device 100 may include, or may be coupled to, an image signal processor 112 for processing image frames from multiple image sensors, such as a first image sensor 101, a second image sensor 102, and a depth sensor 140. The depth sensor 140 may include one or more range imaging devices, such as a time-of-flight (ToF) sensor, a LiDAR (Light Detection and Ranging) system, an infrared (IR) ranging system, a structured light imaging system, and / or a radio frequency (RF) ranging system. In some embodiments, multiple image signal processors may be used instead of a single image signal processor 112, such that each of the image sensors 101 and 102 is coupled to a different image signal processor. In some embodiments, an additional image sensor may share an image signal processor with one of two ISPs coupled to sensors 101 and 102, or an additional image signal processor may be coupled to each of the additional image sensors. In some embodiments, each of the multiple ISPs may perform operations in parallel on image frames acquired from their respective image sensors, such as those described herein with reference to Figures 2 to 6. In some embodiments, one of the multiple ISPs may be a master device and the others may be slave devices, and the master device performs operations on data acquired from two or more of the image sensors, such as coordinate operations and / or operations to determine the disparity between two image frames acquired from two image sensors. A computer vision processor (CVP) may be coupled to each of the image sensors to perform computer vision operations on the output of the image sensors, such as feature matching to determine the difference in the field of view between a first image sensor and a second image sensor, and to provide the difference and / or other feature matching information to the ISP 112 or processor 104, which uses the difference and / or other feature matching information as a disparity value.Feature matching may involve operations such as determining a set of feature points in an image frame acquired from an image sensor, defining the region surrounding each feature point, and optionally normalizing the region surrounding the feature point to improve accuracy, computing local descriptors for the normalized region, and matching descriptors between image frames captured by different image sensors. In some embodiments, feature matching algorithms may include one or more of the following algorithms: scale-invariant feature transform (SIFT), random sample consensus (RANSAC), binary robust independent elementary features (BRIEF), speeded-up robust features (SURF), and / or oriented FAST and rotated brief (ORB).

[0038] In some embodiments, device 100 also includes, or is coupled to, a processor 104 and a memory 106 for storing instructions 108. Device 100 may also include, or be coupled to, a display 114 and several input / output (I / O) components 116, such as a touchscreen interface and / or physical buttons. Device 100 may further include, or be coupled to, a power supply 118 for device 100, such as a battery or components for coupling device 100 to an energy source. Device 100 may also include, or be coupled to, additional features or components not shown. In one example, a wireless interface, which may include several transceivers and a baseband processor, may be included for a wireless communication device. In another example, one or more other sensors (such as a Global Positioning System (GPS) receiver) may be included in, or be coupled to, the device. In a further example, an analog front-end for converting analog image frame data to digital image frame data may be coupled between image sensors 101 and 102 and an image signal processor 112.

[0039] The device may include, or may be coupled to, a sensor hub 150 for interfaceing with sensors to receive data about the movement of device 100, data about the environment surrounding device 100, and / or other non-camera sensor data. Such non-camera sensors may be integrated with and / or coupled to device 100. One exemplary non-camera sensor is a gyroscope, which is a device configured to measure rotation, orientation, and / or angular velocity. Another exemplary non-camera sensor is an accelerometer, which is a device configured to measure acceleration, and the accelerometer may also be used to determine velocity and distance traveled by appropriately integrating the measured acceleration.

[0040] The image signal processor 112 may receive image data, such as that used to form an image frame, from a local bus connection to the image sensors 101 and 102, or from other connections such as a wire interface to an external image sensor or a wireless interface to a distant image sensor. In some embodiments, the device 100 may include a first camera comprising a first image sensor 101 and a corresponding first lens 131, and a second camera comprising a second image sensor 102 and a corresponding second lens 132. Each of the lenses 131 and 132 may have associated autofocus (AF) systems 133 and 134, respectively, which adjust the lenses 131 and 132 to focus on a specific focal plane at a certain scene depth from the sensors 101 and 102. The AF systems 133 and 134 may be supported by the depth sensor 140 and / or may provide depth information to other components of device 100, such as the ISP 112, through metadata associated with image frames captured by sensors 101 and 102. In some embodiments, device 100 may include an interface for receiving image data from image sensors 101 and 102 located away from device 100. Device 100 may perform image processing on image data from combinations of image sensors located within or away from device 100.

[0041] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. For example, the first image sensor 101 and the second image sensor 102 may be included in one multi-camera configuration (such as a dual-camera configuration or triple-camera configuration for a smartphone or other suitable device) or in separate single-camera or multi-camera configurations. The image sensors 101 and 102 may also include, or be coupled to, one or more lenses 131 and 132 for collecting light, one or more apertures for receiving light, one or more shutters for shielding when outside the exposure window, one or more color filter arrays (CFAs) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurements into digital information, or other suitable components for imaging. For example, the first image sensor 101 may be coupled to the first lens 131, and the second image sensor 102 may be coupled to the second lens 132. The first lens 131 and the second lens 132 may have different fields of view, for example, when the first lens 131 is an ultra-wide-angle (UW) lens and the second lens 132 is a wide-angle (W) lens. Device 100 may also include, or be coupled with, a flash, a depth sensor, GPS, or other suitable components for imaging or supporting imaging applications.

[0042] Multiple image sensors may include combinations of ultra-wide-angle (high field of view (FOV)), wide-angle, telephoto, and super-telephoto (low FOV) sensors. That is, each image sensor may be configured through hardware configuration and / or software settings to acquire different but overlapping fields of view. In one configuration, the image sensor consists of different lenses with different magnifications that result in different fields of view. The sensors may be configured such that the UW sensor has a larger FOV than the W sensor, the W sensor has a larger FOV than the T sensor, and the T sensor has a larger FOV than the UT sensor. For example, a sensor configured for a wide-angle FOV can capture a field of view in the range of 64-84 degrees, a sensor configured for an ultra-side FOV can capture a field of view in the range of 100-140 degrees, a sensor configured for a telephoto FOV can capture a field of view in the range of 10-30 degrees, and a sensor configured for a super-telephoto FOV can capture a field of view in the range of 1-8 degrees. Some aspects of this disclosure include processing captured image frames, such as adjusting the spatial alignment of one or more of the captured image frames, while the device transitions from capturing an image of a scene using a first of a plurality of image sensors to capturing an image of a scene using a second of a plurality of image sensors. Aspects of this disclosure may be used to capture image frames using multiple image sensors of an image capture device. For example, camera motion data used for digital image stabilization (DIS) may be obtained using a combination of image frames from multiple image sensors.

[0043] The image signal processor 112 processes image frames captured by image sensors 101 and 102. Figure 1 shows device 100 including two image sensors 101 and 102 coupled to the image signal processor 112, but any number of image sensors may be coupled to the image signal processor 112. In addition, any number of additional image sensors or image signal processors may be present for device 100. In some embodiments, the image signal processor 112 may execute instructions from memory, such as instruction 108 from memory 106, instructions stored in separate memories coupled to or included in the image signal processor 112, or instructions provided by processor 104. In addition, or alternatively, the image signal processor 112 may run software and / or include certain hardware (such as one or more integrated circuits (ICs)) to perform one or more operations described herein.

[0044] In some implementations, memory 106 may include a non-transient or non-transitory computer-readable medium for storing computer-executable instructions 108 for performing all or part of one or more operations described in this disclosure. In some implementations, instructions 108 include a camera application (or other suitable application) to be executed by device 100 to generate an image or video. Instructions 108 may also include other applications or programs executed by device 100, such as an operating system or specific applications other than those for image or video generation. Execution of the camera application by a processor 104, for example, may cause device 100 to generate an image using image sensors 101 and 102 and an image signal processor 112. Memory 106 may also be accessed by the image signal processor 112 to store processed frames or by the processor 104 to retrieve processed frames. In some embodiments, device 100 does not include memory 106. For example, device 100 may be a circuit including an image signal processor 112, and memory may be located outside of device 100. Device 100 may be coupled to memory and configured to access memory to write output frames for display or long-term storage.

[0045] In some embodiments, the processor 104 may include one or more general-purpose processors capable of executing scripts or instructions of one or more software programs, such as instructions 108 stored in memory 106. For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in memory 106. When executing the camera application, the processor 104 may be configured to instruct an image signal processor 112 to perform one or more operations relating to the image sensor 101 or 102. For example, the camera application may receive a capture command, upon receiving the capture command, a video containing a sequence of image frames is captured and processed by one or more methods described herein for reducing artifacts by warping some of the image frames. The execution of instructions 108 by the processor 104 outside of the camera application may also cause the device 100 to perform any number of functions or operations. In some embodiments, the processor 104 may include ICs or other hardware in addition to its ability to execute software causing the device 100 to perform some functions or operations, such as the operations described herein. In some other embodiments, the device 100 does not include the processor 104, for example, when all of the functions described are configured in the image signal processor 112.

[0046] In some embodiments, at least one of the image signal processor 112 or processor 104 can execute instructions to perform various operations described herein. For example, the execution of an instruction may instruct the image signal processor 112 to start or stop capturing an image frame or a sequence of image frames. As another example, the execution of an instruction may instruct the image signal processor 112 to switch from capturing a first image of a scene captured using a first image sensor 101 to capturing a second image of a scene captured using a second image sensor 102. As yet another example, the execution of an instruction may instruct the image signal processor 112 to change the zoom level from a first zoom level to a second zoom level, which may result in a switch from a first image sensor having a first field of view to a second image sensor having a second field of view. As yet another example, the execution of an instruction may instruct the image signal processor 112 to switch from a first image sensor for capturing image frames to a second image sensor for capturing image frames while recording a sequence of image frames.

[0047] In some embodiments, the display 114 may include one or more suitable displays or screens that enable user interaction and / or the presentation of items to the user, such as previews of image frames captured by image sensors 101 and 102. In some embodiments, the display 114 is a touch-sensitive display. The I / O component 116 may be or include any suitable mechanism, interface, or device for receiving input (such as commands) from the user and providing output to the user. For example, the I / O component 116 may include (but not limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a compressible bezel, one or more buttons (such as a power button), sliders, switches, and the like.

[0048] Although shown coupled to each other via processor 104, processor 104, memory 106, image signal processor 112, display 114, and I / O components 116 may be coupled to each other in various other configurations, such as via one or more local buses not shown for simplicity. Although shown separately from processor 104, image signal processor 112 may be a core of processor 104, or an application processor unit (APU) included in a system-on-a-chip (SoC) or otherwise included in processor 104. Device 100 is mentioned in the examples herein for performing aspects of the disclosure, but some device components may not be shown in Figure 1 to avoid obscuring aspects of the disclosure. In addition, other components, numerous components, or combinations of components may be included in a suitable device for performing aspects of the disclosure. Thus, the disclosure is not limited to a specific device or component configuration including device 100.

[0049] A multi-sensor device such as device 100 can switch from one image sensor to another, such as between sensors 101 and 102, based on user requests or when certain criteria are met. For example, a user may request to switch from the wide-angle (W) lens corresponding to sensor 101 to the telephoto (T) lens corresponding to sensor 102 by activating portrait mode in a camera application that accesses sensors 101 and 102. As another example, a user may change the zoom level in a camera application, which causes the image signal processor 112 to switch from sensor 101 to sensor 102 based on the characteristics of the lenses corresponding to sensors 101 and 102. As yet another example, changing scene characteristics such as light level may cause the image signal processor 112 to switch from sensor 101 to sensor 102 to achieve better light sensitivity.

[0050] In any of these exemplary (or other) switching between image sensors, artifacts may appear in the sequence of image frames at or around the time of the switch between sensors due to differences in the sensor fields of view. The drawbacks described herein are representative only and are included to highlight the problems that the inventors have identified with respect to existing devices and have sought to improve. The embodiments of the device described below may address some or all of the drawbacks, as well as others known in the art. The improved embodiments of the device described herein may offer benefits other than those described above and may be used in applications other than those described above.

[0051] In one embodiment of device 100, image frames captured from sensors 101 and 102 can be corrected to improve the alignment of the field of view between sensors 101 and 102. For example, an input image frame from one sensor can be geometrically warped based on the parallax, or difference, between image frames captured by sensors 101 and 102. The parallax between image frames may arise from the different positions of image sensors 101 and 102 and from imperfect alignment between sensors 101 and 102. The parallax between image frames can be determined and used as a basis for determining the amount and method of geometric warping to be applied during the generation of the corrected image frame. The actual parallax between image frames can be determined by feature matching between a first image frame captured from a first image sensor and a second image frame captured from a second image sensor. Feature matching can be performed using computer vision (CV).

[0052] The feature matching process may fail due to sensor failure, scene conditions, scene changes, and / or other factors. When feature matching fails, the disparity values ​​for determining geometric warping are unavailable or inaccurate, which can lead to poor performance of geometric warping. As a result, the sequence of image frames may be accompanied by significant artifacts, or in the worst case, image frames may be missing from the sequence. A model may be constructed from known disparity values, such as those determined during the device's calibration routine and / or preloaded on the device, during a previous image capture on the device. This model may be used to supply predicted disparity values ​​to complement or replace the disparity values ​​determined by feature matching or other disparity determination techniques. Model generation is described in one embodiment in Figure 2.

[0053] Figure 2 is a block diagram illustrating the modeling of image sensor disparity in a multi-sensor device according to one or more embodiments of the present disclosure. System 200 shows the generation of model 212 in the disparity predictor 210. A computer vision (CV) processor (CVP) 202 receives first and second image frames capturing scenes from different fields of view. The image frames may be received by the CV processor 202 from the image sensor via an image signal processor 112, etc., or retrieved from memory. The CV processor 202 may analyze the images to identify image features and correlate matching image features from the first image frame to the second image frame. Based on the matching features, the CV processor 202 may determine a shift of features from the first image frame to the second image frame. This shift may be represented as one or more disparity values, such as a first disparity value along a first axis and a second disparity value along a second axis. In some embodiments, the first and second axes may be orthogonal, for example, when the first and second axes are the x and y axes. Determined disparity values, such as dx and dy values, may be provided to the disparity predictor 210 by the CV processor 202. The predictor 210 may also receive scene depth corresponding to the scene reflected in the first and second image frames from which the dx and dy values ​​are determined. Scene depth may reflect the distance from the image sensor to the object of interest, measured by a depth sensor. Scene depth may, alternatively or additionally, be received as the autofocus distance corresponding to the first or second image frame from the autofocus sensor. The predictor 210 may apply the scene depth and disparity values ​​to train the model 212. The operation of the disparity predictor 210 may be performed by an image signal processor (ISP), such as ISP 112. Alternatively, the operation of the disparity predictor 210 may be performed by a digital signal processor (DSP) or a processor, such as processor 104.

[0054] Model 212 may be a nonlinear regression model, a linear regression model, or a machine learning algorithm. The machine learning model may, in some embodiments as described herein, include logistic regression techniques, linear discriminant analysis, linear regression analysis, artificial neural networks, machine learning classifier algorithms, or classification / regression trees. In some embodiments, machine learning may include one or more artificial neural networks, which may include interconnected groups of artificial neurons (e.g., neuron models) for modeling relationships between parameters such as disparity and scene depth. In some embodiments, machine learning may include one or more convolutional neural networks, which are a type of feedforward artificial neural network. A convolutional neural network may include a collection of neurons, each having a receptive field and tiling the input space together. In some embodiments, machine learning may include one or more deep learning architectures, such as deep belief networks and deep convolutional networks, which are hierarchical neural network architectures, where the output of the first layer of neurons becomes the input to the second layer of neurons, the output of the second layer of neurons becomes the input to the third layer of neurons, and so on. Deep neural networks may be trained to recognize a hierarchy of features. In various embodiments, machine learning systems may employ some kind of Naive Bayesian predictive modeling analysis, learning vector quantization, or implementations of boosting algorithms such as Adaboost or stochastic gradient boosting systems to iteratively update weights to train machine learning classifiers to determine the relationships between influencing attributes such as scene depth and disparity, and / or the extent to which such influencing attributes affect the outcome of such systems or disparity values.

[0055] After training, the disparity predictor 210 may use Model 212 to predict disparity values ​​for pairs of image frames captured from the image device 100. In some embodiments, the display predictor 210 may be run by the logic circuits of device 100 to obtain training of Model 212 based on image frames captured by image sensors 101 and 102. In some embodiments, the display predictor 210 may be run by a remote cloud-based computing system. When Model 212 is generated in the cloud, it may be exported and used on one or more image capture devices, such as device 100. In some embodiments, Model 212 may be loaded on individual devices 100 and then trained in the cloud from data captured from other image capture devices to generate a general-purpose model that is revised by predictors on individual devices based on the unique characteristics of each device. For example, dropping a device may result in a change in disparity between image sensors, which is specific to a particular device and may be taken into consideration in the local revision of Model 212.

[0056] Although the CV processor 202 is shown to generate disparity values ​​for training model 212, other techniques may be used to generate the training data. Regardless of the method used to train model 212, the disparity predictor 210 may use the estimator 214 to generate predicted disparity values ​​using model 212. An exemplary generation and use of predicted disparity values ​​is shown in Figure 3. Figure 3 is a block diagram showing warping of image frames using predicted disparity according to one or more embodiments of the present disclosure. System 300 uses the disparity predictor 210 to generate one or more predicted disparity values. The estimator 214 of the predictor 210 may receive a scene depth value for which the predicted disparity values ​​are desired. For example, the autofocus depth corresponding to the image frame being processed may be input to the estimator 214. The estimator 214 may access model 212 to obtain predicted disparity values ​​dx and dy for the input scene depth.

[0057] Predicted disparity values ​​may be input to the geometric warp module 310 to correct the input image frame. The warp module 310 receives the input image frame and the predicted disparity values. The warp module 310 then determines a corrected image frame 330 based at least partially on the predicted disparity values ​​and the input image frame. The corrected image frame 330 may be further processed before, after, or in parallel with the processing of the warp module 310. Other processing of the corrected image frame may include computational photography, high dynamic range (HDR), multi-frame noise reduction (MFNR), low-light enhancement, and super-resolution processing. Such computational photography may include processing in which the determination of the corrected image frame includes merging the input image frame (after applying warping by a transformation matrix to match with another image sensor) with image frames from another image sensor and / or additional image frames from the same image sensor to improve the appearance of the corrected image frame. Examples of such computational photography include obtaining an improved dynamic range from the fusion of one or more additional image frames from one or more other image sensors, obtaining reduced noise from the fusion of one or more additional image frames from one or more image sensors, and / or obtaining higher resolution from the fusion of one or more additional image frames from one or more other image sensors. The corrected image frame 330 may be geometrically warped to improve the alignment of the field of view of the input image frame to other image frames in a sequence of image frames 340 captured at or around the time of switching from the first image sensor to a second image sensor having a different field of view from the first image sensor. The corrected image frame 330 may appear in a video sequence of image frames between a first image frame captured from the first image sensor and a second image frame captured from the second image sensor.

[0058] An exemplary model 212 for generating predicted disparity values ​​by the estimator 214 is shown in Figure 4. Figure 4 is a graph showing a generated model for image sensor disparity according to one or more aspects of the present disclosure. Graph 400 includes a first line 402 reflecting a first predicted disparity value and a second line 404 reflecting a second predicted disparity value. The first and second lines 402 and 404 may reflect disparity values ​​along different axes, such as the dx disparity value and dy disparity value described above. The first line 402 may be generated from data 412 collected during training of model 212, and the second line 404 may be generated from data 414 collected during training of model 212.

[0059] Individual data points for data 412 and 414 may be stored in memory within the predictor 210, in memory 106, in the cloud, or in other storage locations. In some embodiments, data 412 and 414 may be used to update the training of model 212, for example, when new data is received. In some embodiments, the model is generated on demand from the stored data 412 and 414 when predicted disparity values ​​are requested. In some embodiments, the amount of data stored is a predetermined number of sets of disparity values ​​having corresponding scene depths. When new data is received, older sets may be discarded based on algorithms such as first-in, first-out (FIFO) or allocation priority associated with the set of values. In some embodiments, the amount of data stored is limited to a certain time period such that sets of data exceeding a certain lifetime are removed from the stored data. In some embodiments, a combination of time, number, and other criteria may be used to determine the lifetime of the stored data 412 and 414 to generate model 212. For example, data from the accelerometer may be used to determine if a drop incident has occurred, which could cause Model 212 to become inaccurate. As a result, data received before the drop incident should be removed or unweighted during the generation of Model 212. As another example, a certain number of disparity samples for different depth ranges may be stored, such as 10 disparity values ​​for depths of 0-10 centimeters, another 10 disparity values ​​for depths of 10-20 centimeters, another 10 disparity values ​​for depths of 20-50 centimeters, and 20 values ​​for depths from 50 centimeters to infinity. Model 212 may be updated by least squares whenever new data is added, old data is aged out, and / or old data is replaced with new data. Model 212, reflecting lines 402 and 404, may be represented by values ​​such as coefficients corresponding to the parameters of equations and / or matrices and / or machine learning algorithms, as well as their values ​​stored in the predictor 210.

[0060] An embodiment of a method for generating and applying an embodiment of Model 212 described herein for correcting an image frame will be described with reference to Figure 5. Figure 5 is a flowchart of a method for correcting an image frame using predictive disparity data according to one or more embodiments of the present disclosure. Method 500 begins in block 502, where a first disparity value is received, indicating the difference in the field of view between a first image sensor and a second image sensor at a corresponding first depth. In block 504, a second disparity value is received, indicating the difference in the field of view between the first image sensor and the second image sensor at a corresponding second depth. The model can be constructed from just two scene depths having corresponding disparity values.

[0061] In block 506, a model for the disparity between a first image sensor and a second image sensor is determined based on a first disparity value, a first depth, a second disparity value, and a second depth. The first disparity value and the first depth may form a first training set, and the second disparity and the second depth may form a second training set. The training sets may include disparity values ​​having the same depth value. The model may have a machine learning algorithm that learns the relationship between disparity and scene depth from the training sets. Alternatively, the model may be a programmed equation having coefficients, constants, and / or other parameters that are adjusted when a training set is received to reflect the relationship between disparity and scene depth. Thus, the model can be constructed with just two training sets, for example, when the model is a linear equation. The model can be reconstructed for higher-order equations when additional training sets are available.

[0062] In block 508, the model is applied to generate predicted disparity values ​​for an input image frame. For example, depths corresponding to the input image frame may be input to the model to generate one or more predicted disparity values ​​that predict the disparity value for the difference in the field of view between a first image sensor and a second image sensor. In some embodiments, the input image frame may be an image frame captured by either the first or second image sensor. In some embodiments, the input image frame may be an image frame captured by an image sensor corresponding to either the first or second image sensor, such as a similar sensor on a different device from the device on which the training data for the model was acquired. Predicted disparity values ​​are generated for an image frame captured from either the first and / or second image sensor and may be compared to the actual disparity measured from the image sensor to provide feedback for adjusting the model. The predicted disparity values ​​may be used to correct an image frame captured from either or both of the first and second image sensors. In one exemplary application of the predicted disparity values, the predictions may be used to geometrically warp either or both of the first and second image sensors.

[0063] In block 510, the input image frame is warped based on the predicted disparity value in block 508. Warping may involve transforming the grid of the input image frame to align with the field of view of the other image sensor during image sensor switching. In one embodiment, the predicted disparity value may be used to generate an artificial computer vision processing matrix to replace the output of the CV processor. In one embodiment, the predicted disparity value may be applied on the crop window to achieve off-center cropping to reduce the shifts that need to be performed in the computer vision processing stage. When the warped image frame is viewed as a video in a sequence of image frames, the warped image frame may reduce the appearance of jumps, skips, or other artifacts around the switching of image sensors. Image warping may continue across two or more image frames in a video sequence to smooth the transition from one image sensor to another, for example, by scaling the geometric warping with a scaling factor that decreases toward a unit value for each subsequent frame. When the scaling factor reaches a unit value, and as a result the warped image frame based on the current image sensor matches the field of view of another image sensor, the image sensor can switch to that other image sensor. After the switch, processing of image frames from the other image sensor may continue, and the preview video or recorded video continues. Warping is described as a method for determining a corrected image frame from the input image frame, but other processing of the input image frame may be performed in addition to or as an alternative to warping. Part of the processing may be based on predicted disparity values, and part of the processing may be based on other values.

[0064] In some embodiments, predicted disparity values ​​may be used to correct all image frames output from an image signal processor. In some embodiments, predicted disparity values ​​may be selectively used during the operation of an image capture device to determine which output frames to correct. For example, the use of predicted disparity values ​​may be turned on for a certain period of time and off for another period of time. The "off" period may be used to train a model, and then the predicted disparity values ​​may be turned on again. For example, the user may enter a calibration mode, in which the device requires the user to enter a well-lit room and take a series of photographs of an object at some depth by walking towards the object to obtain a training set. As another example, criteria for input image frames may be evaluated, and the evaluation of those criteria was used to determine whether to use predicted disparity values ​​generated from a model for any particular input image frame. One embodiment for selectively applying predicted disparity values ​​in determining the corrected image frame will be described with reference to Figure 6.

[0065] Figure 6 is a flowchart illustrating one or more methods for selectively using predictive disparity data in correcting image frames. Method 600 includes receiving a first input image frame and a second input image frame in block 602. In block 604, image features in one or both of the first and second input image frames are analyzed. The analysis in block 604 may include feature matching between the first and second input image frames. The difference between the locations of the matched features from the first and second image frames may be used as the actual disparity in warping the image frames. However, image matching may fail, leading to subsequent failures in image warping, which can cause artifacts when the image frames are viewed as a video in a sequence of image frames. The analysis in block 604 may be considered to determine whether to use the actual disparity and / or predictive disparity in processing the input image frames.

[0066] In block 606, criteria are evaluated to determine whether predictive disparity should be used to determine the corrected image frame. The criteria may include whether feature matching was successful or unsuccessful. The criteria may also include, or alternatively, whether the confidence level of feature matching is above a threshold level. Other criteria based on the characteristics of the image frame may also be used. For example, to determine whether the image is too dark for high confidence in image matching, the average brightness of the first and / or second input image frame may be determined and compared to a threshold. As another example, to determine whether the image is too out of focus for high confidence in image matching, the focal position and / or confidence of the lens capturing the first and / or second image frame may be determined and compared to a threshold. In some embodiments, feature matching in block 604 may generate a transformation matrix for the input image frame to warp the input image frame to the field of view of the first or second image sensor, the other of the two. To evaluate whether predictive disparity should be used, the confidence level of the transformation matrix may be compared to a threshold level.

[0067] If the criteria are met in block 606, method 600 proceeds to block 608, which determines the predicted disparity at the scene depth corresponding to one or both of the first and second input image frames. Then, in block 610, based on the predicted disparity in block 608, the first input image frame may be warped to the second input image frame, or the second input image frame may be warped to the first input image frame.

[0068] If the criteria for using predicted disparity are not met in block 606, method 600 proceeds to block 612, in which feature matching is used to determine the disparity for the first and second input image frames. Then, in block 614, based on the disparity determined in block 612, the first input image frame may be warped to the second input image frame, or the second input image frame may be warped to the first input image frame.

[0069] In one or more embodiments, techniques for supporting image capture with reduced artifacts such as jumping and skipping of objects in a scene recorded in a video sequence, resulting in higher quality photographs and videos produced by the device. The improved video quality may be provided by obtaining more accurate parallax values ​​while warping image frames captured by one image sensor to align them with the field of view of another image sensor, in order to facilitate switching from one image sensor to another. In addition, the device may perform one or more embodiments, or operate according to one or more embodiments, as described below. In some implementations, the device includes a wireless device such as a UE. In some implementations, the device may include at least one processor and memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the device. In some other implementations, the device may include a non-temporary computer-readable medium recording program code, which may be computer-executable to cause the computer to perform the operations described herein with respect to the device. In some implementations, the device may include one or more means configured to perform the operations described herein. In some implementations, the wireless communication method may include one or more operations described herein with respect to the device.

[0070] In one or more embodiments, a technique for supporting image capture with reduced artifacts such as jumping and skipping of objects in a scene recorded in a video sequence, resulting in higher quality photos and videos produced by the device. The improved video quality may be provided by obtaining more accurate parallax values ​​while warping image frames captured by one image sensor to align them with the field of view of another image sensor, in order to facilitate switching from one image sensor to another. In one or more embodiments, supporting image capture may include a device configured to receive a first parallax value indicating the difference in the field of view between a first image sensor and a second image sensor. The device is further configured to receive a first depth value corresponding to the first parallax value. The device is further configured to determine a model for the parallax between the first image sensor and the second image sensor at a plurality of depth values, wherein the model is determined to be at least partially based on the first parallax value and the first depth. In addition, the device may perform one or more embodiments, or operate according to one or more embodiments, as described below. In some implementations, the device includes a wireless device such as a UE. In some implementations, the device may include at least one processor and memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the device. In some other implementations, the device may include a non-temporary computer-readable medium recording program code, which may be executable by a computer to cause the computer to perform the operations described herein with respect to the device. In some implementations, the device may include one or more means configured to perform the operations described herein. In some implementations, a wireless communication method may include one or more operations described herein with respect to the device, and may include transmitting a corrected image frame to another mobile device, to a base station, through a base station, or directly to a server or another mobile device.

[0071] In the first embodiment, the device is configured to receive an input image frame from either a first image sensor or a second image sensor.

[0072] In a second embodiment, in combination with the first embodiment, the device is configured to receive input image depths corresponding to input image frames.

[0073] In a third embodiment, in combination with the second embodiment, the device is configured to determine a predicted disparity value corresponding to the input image depth based at least in part on a model.

[0074] In a fourth embodiment, in combination with the third embodiment, the apparatus is configured to determine a corrected image frame based at least in part on the input image frame and the predicted disparity value.

[0075] In the fifth aspect, determining a corrected image frame in combination with one or more of the first to fourth aspects includes determining a corrected image frame as warped to match the field of view of the first or second image sensor, based on a predicted disparity value.

[0076] In the sixth aspect, determining a corrected image frame in combination with one or more of the first to fifth aspects includes determining a transformation matrix for warping the input image frame to the field of view of the first or second image sensor.

[0077] In the seventh aspect, in combination with the sixth aspect, determining the transformation matrix includes determining the transformation matrix using computer vision processing (CVP).

[0078] In the eighth aspect, in combination with the sixth aspect, the apparatus is configured to determine whether the confidence level associated with the transformation matrix is ​​below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the confidence level of the transformation matrix is ​​below a threshold level.

[0079] In the ninth aspect, in combination with one or more of the first to eighth aspects, the apparatus is configured to determine that the image characteristics of an input image frame are below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the image characteristics are below a threshold level.

[0080] In the tenth embodiment, in combination with one or more of the first to ninth embodiments, the apparatus is configured to determine that the luminance of an input image frame is below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the luminance is below a threshold level.

[0081] In the eleventh embodiment, in combination with one or more of the first to tenth embodiments, the apparatus is configured to determine a video sequence including a first image frame from a first image sensor, a second image frame from a second image sensor, and a correction image frame. The correction image frame appears in the video sequence between the first image frame and the second image frame.

[0082] In the twelfth embodiment, the step of determining a corrected image frame, in combination with one or more of the first to eleventh embodiments, is further based on the image frame of the other of the first or second image sensor.

[0083] In the 13th aspect, either alone or in combination with one or more of the first to 12th aspects, the first disparity value represents the difference in the field of view along the first axis.

[0084] In a fourteenth aspect, in combination with the thirteenth aspect, the method is configured to receive a second disparity value indicating the difference in the field of view between a first image sensor and a second image sensor along a second axis different from the first axis. The step of determining the model is further at least in part based on the second disparity value.

[0085] In the 15th aspect, determining the model, either alone or in combination with one or more of the 1st to 14th aspects, includes storing a plurality of disparity values. The model is based on the plurality of disparity values.

[0086] In the sixteenth aspect, in combination with the fifteenth aspect, the previous value among the multiple disparity values ​​is replaced with a first disparity value based on at least one of the times associated with some of the multiple disparity values ​​or the previous value.

[0087] In the 17th embodiment, either alone or in combination with one or more of the first to 16th embodiments, the first depth value includes an autofocus depth corresponding to a first input image frame captured by a first image sensor.

[0088] In the 18th embodiment, the apparatus is configured, either alone or in combination with one or more of the first to 17th embodiments, to determine a first depth value based on range imaging.

[0089] In the 19th aspect, in combination with the 18th aspect, determining a first depth value includes determining a first depth value based on a time-of-flight (ToF) measurement.

[0090] In the 20th aspect, determining a first depth value in combination with one or more of the 18th to 19th aspects includes determining a first depth value based on light detection and ranging (LIDAR) measurements.

[0091] In one or more embodiments, the technique for supporting image capture may include additional embodiments, such as any single embodiment or any combination of embodiments, as described below or elsewhere in this specification with respect to one or more other processes or devices. In one or more embodiments, supporting image capture may include a device configured for a processor. The device is a memory coupled to the processor, further configured for storing instructions, when executed by the processor, that cause the device to perform an action including receiving a first disparity value indicating the difference in the field of view between a first image sensor and a second image sensor. The device is further configured to receive a first depth value corresponding to the first disparity value. The device is further configured to determine a model for the disparity between the first image sensor and the second image sensor at a plurality of depth values, wherein the model is determined to be at least partially based on the first disparity value and the first depth. In addition, the device may perform one or more embodiments or operate according to one or more embodiments, as described below. In some implementations, the device includes a wireless device such as a base station. In some implementations, the device may include at least one processor and memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the device. In some other implementations, the device may include a non-temporary computer-readable medium recording program code, which may be executable by a computer to cause the computer to perform the operations described herein with respect to the device. In some implementations, the device may include one or more means configured to perform the operations described herein. In some implementations, a wireless communication method may include one or more operations described herein with respect to the device.

[0092] In the 21st aspect, the command causes the device to perform an operation that includes receiving an input image frame from either the first or second image sensor.

[0093] In the 22nd aspect, in combination with the 21st aspect, the device is configured to receive an input image depth corresponding to an input image frame.

[0094] In the 23rd aspect, in combination with the 22nd aspect, the apparatus is configured to determine a predicted disparity value corresponding to the input image depth based at least in part on a model.

[0095] In the 24th aspect, in combination with the 23rd aspect, the apparatus is configured to determine a corrected image frame based at least in part on an input image frame and a predicted disparity value.

[0096] In the 25th aspect, determining a corrected image frame in combination with one or more of the 21st to 24th aspects includes determining a corrected image frame that has been warped to match the field of view of the first or second image sensor, based on a predicted disparity value.

[0097] In the 26th aspect, determining a corrected image frame in combination with one or more of the 21st to 25th aspects includes determining a corrected image frame that has been warped to match the field of view of the first or second image sensor, based on a predicted disparity value.

[0098] In the 27th aspect, determining a corrected image frame in combination with one or more of the 21st to 26th aspects includes determining a transformation matrix for warping the input image frame to the field of view of the first or second image sensor.

[0099] In the 28th aspect, in combination with the 27th aspect, determining the transformation matrix includes determining the transformation matrix using computer vision processing (CVP).

[0100] In the 29th aspect, in combination with one or more of the 27th to 28th aspects, the instruction causes the device to perform an operation that further includes determining whether the confidence level associated with the transformation matrix is ​​below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the confidence level of the transformation matrix is ​​below a threshold level.

[0101] In the 30th aspect, in combination with one or more of the 21st to 29th aspects, the instruction causes the device to perform an operation that further includes determining that the image characteristics of an input image frame are below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the image characteristics are below a threshold level.

[0102] In the 31st aspect, in combination with one or more of the 21st to 30th aspects, the instruction causes the device to perform an operation that further includes determining that the brightness of an input image frame is below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the brightness is below a threshold level.

[0103] In the 32nd aspect, in combination with one or more of the 21st to 31st aspects, the command causes the device to perform an operation that further includes determining a video sequence comprising a first image frame from a first image sensor, a second image frame from a second image sensor, and a corrected image frame. The corrected image frame appears in the video sequence between the first image frame and the second image frame.

[0104] In the 33rd aspect, in combination with one or more of the 21st to 32nd aspects, the step of determining a corrected image frame is further based on the image frame of the other of the first or second image sensor.

[0105] In the 34th aspect, either alone or in combination with one or more of the 21st to 34th aspects, the first disparity value represents the difference in the field of view along the first axis.

[0106] In the 35th aspect, in combination with the 33rd aspect, the instruction causes the device to perform an operation that further includes receiving a second disparity value indicating the difference in the field of view between a first image sensor and a second image sensor along a second axis different from the first axis. The step of determining the model is further at least in part based on the second disparity value.

[0107] In the 36th aspect, determining the model, either alone or in combination with one or more of the 21st to 35th aspects, includes storing multiple disparity values. The model is based on the multiple disparity values.

[0108] In the 37th aspect, in combination with the 36th aspect, the previous value among the multiple disparity values ​​is replaced with a first disparity value based on at least one of the times associated with some of the multiple disparity values ​​or the previous value.

[0109] In the 38th aspect, either alone or in combination with one or more of the 21st to 37th aspects, the first depth value includes an autofocus depth corresponding to a first input image frame captured by a first image sensor.

[0110] In the 39th aspect, either alone or in combination with one or more of the 21st to 38th aspects, the command causes the device to perform an operation that further includes determining a first depth value based on range imaging.

[0111] In the 40th aspect, in combination with the 39th aspect, determining the first depth value includes determining the first depth value based on time-of-flight (ToF) measurements.

[0112] In the 41st aspect, in combination with the 40th aspect, determining a first depth value includes determining a first depth value based on light detection and ranging (LIDAR) measurements.

[0113] In one or more embodiments, the technique for supporting image capture may include additional embodiments, such as any single embodiment or any combination of embodiments, described in relation to one or more other processes or devices described below or elsewhere in this specification. In one or more embodiments, supporting image capture may include an apparatus configured to cause a device to perform an operation including receiving a first disparity value indicating the difference in the field of view between a first image sensor and a second image sensor. The apparatus may be further configured to perform an operation including receiving a first depth value corresponding to the first disparity value. The apparatus may be further configured to perform an operation including determining a model for the disparity between the first image sensor and the second image sensor at a plurality of depth values, wherein the model is determined to be at least partially based on the first disparity value and the first depth. In addition, the apparatus may perform one or more embodiments or operate according to one or more embodiments, as described below. In some implementations, the apparatus includes a wireless device such as a base station. In some implementations, the apparatus may include at least one processor and memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the device. In some other implementations, the device may include a non-temporary computer-readable medium recording program code, which may be executable by a computer to cause the computer to perform the operations described herein with respect to the device. In some implementations, the device may include one or more means configured to perform the operations described herein. In some implementations, a wireless communication method may include one or more operations described herein with respect to the device.

[0114] In the 42nd aspect, when the instruction is executed by the device's processor, the device further performs an operation which includes receiving an input image frame from either the first or second image sensor.

[0115] In the 43rd aspect, in combination with the 42nd aspect, the input image depth corresponding to the input image frame is received.

[0116] In the 44th aspect, in combination with the 43rd aspect, a predicted disparity value corresponding to the input image depth is determined based at least partially on the model.

[0117] In the 45th aspect, in combination with the 44th aspect, a corrected image frame is determined at least partially based on the input image frame and the predicted disparity value.

[0118] In the 46th aspect, determining a corrected image frame in combination with one or more of the aspects from the 42nd to the 45th aspects includes determining a corrected image frame that has been warped to match the field of view of the first or second image sensor, based on a predicted disparity value.

[0119] In the 47th aspect, determining a corrected image frame in combination with one or more of the 42nd to 46th aspects includes determining a transformation matrix for warping the input image frame to the field of view of the first or second image sensor.

[0120] In the 48th aspect, in combination with the 47th aspect, determining the transformation matrix includes determining the transformation matrix using computer vision processing (CVP).

[0121] In the 49th aspect, in combination with one or more of the 47th to 48th aspects, the instruction, when executed by the device's processor, causes the device to perform an operation that further includes determining whether the confidence level associated with the transformation matrix is ​​below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the confidence level of the transformation matrix is ​​below a threshold level.

[0122] In the 50th aspect, in combination with one or more of the 42nd to 49th aspects, the instruction, when executed by the device's processor, causes the device to perform an operation that further includes determining that the image characteristics of an input image frame are below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the image characteristics are below a threshold level.

[0123] In the 51st aspect, in combination with one or more of the 42nd to 50th aspects, the instruction, when executed by the device's processor, causes the device to perform an operation that further includes determining that the luminance of an input image frame is below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the luminance is below a threshold level.

[0124] In the 52nd aspect, in combination with one or more of the 42nd to 51st aspects, the instruction, when executed by the device's processor, causes the device to perform an operation that further includes determining a video sequence comprising a first image frame from a first image sensor, a second image frame from a second image sensor, and a corrected image frame. The corrected image frame appears in the video sequence between the first image frame and the second image frame.

[0125] In the 53rd aspect, in combination with one or more of the 42nd to 52nd aspects, the step of determining a corrected image frame is further based on the image frame of the other of the first or second image sensor.

[0126] In the 54th aspect, either alone or in combination with one or more of the 42nd to 53rd aspects, the first disparity value represents the difference in the field of view along the first axis.

[0127] In the 55th aspect, in combination with the 54th aspect, the instruction, when executed by the device's processor, further includes causing the device to perform an operation that indicates the difference in the field of view between a first image sensor and a second image sensor along a second axis different from the first axis. The step of determining the model is further at least in part based on the second disparity value.

[0128] In the 56th aspect, determining a model, either alone or in combination with one or more of the 42nd to 55th aspects, includes storing multiple disparity values. The model is based on the multiple disparity values.

[0129] In the 57th aspect, in combination with the 56th aspect, the previous value among the multiple disparity values ​​is replaced with a first disparity value based on at least one of the times associated with some of the multiple disparity values ​​or the previous value.

[0130] In the 58th aspect, either alone or in combination with one or more of the 42nd to 57th aspects, the first depth value includes an autofocus depth corresponding to a first input image frame captured by a first image sensor.

[0131] In the 59th aspect, either alone or in combination with one or more of the 42nd to 58th aspects, the instruction, when executed by the device's processor, causes the device to perform an operation that further includes determining a first depth value based on range imaging.

[0132] In the 60th aspect, in combination with the 59th aspect, determining a first depth value includes determining a first depth value based on a time-of-flight (ToF) measurement.

[0133] In the 61st aspect, determining a first depth value in combination with one or more of the aspects of the 59th to 60th aspects includes determining a first depth value based on light detection and ranging (LIDAR) measurements.

[0134] In one or more embodiments, the technique for supporting image capture may include additional embodiments, such as any single embodiment or any combination of embodiments, described in relation to one or more other processes or devices described below or elsewhere in this specification. In one or more embodiments, supporting image capture may include an apparatus configured for a first image sensor comprising a first field of view, a second image sensor comprising a second field of view at least partially overlapping with the first field of view, a processor coupled to the first and second image sensors, and memory coupled to the processor. The apparatus is further configured to perform a step including receiving a first disparity value indicating the difference between the fields of view between the first and second image sensors. The apparatus is further configured to receive a first depth value corresponding to the first disparity value. The apparatus is further configured to determine a model for the disparity between the first and second image sensors at a plurality of depth values, wherein the model is determined to be at least partially based on the first disparity value and the first depth. In addition, the device may perform one or more modes, or operate according to one or more modes, as described below. In some implementations, the device includes a wireless device such as a base station. In some implementations, the device may include at least one processor and memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the device. In some other implementations, the device may include a non-temporary computer-readable medium recording program code, the program code may be executable by a computer to cause the computer to perform the operations described herein with respect to the device. In some implementations, the device may include one or more means configured to perform the operations described herein. In some implementations, a wireless communication method may include one or more operations described herein with respect to the device.

[0135] In the 62nd aspect, the processor is further configured to perform a step which includes receiving an input image frame from either a first image sensor or a second image sensor.

[0136] In the 63rd aspect, in combination with the 62nd aspect, the input image depth corresponding to the input image frame is received.

[0137] In the 64th aspect, in combination with the 63rd aspect, a predicted disparity value corresponding to the input image depth is determined based at least partially on the model.

[0138] In the 65th aspect, in combination with the 64th aspect, a corrected image frame is determined at least partially based on the input image frame and the predicted disparity value.

[0139] In the 66th aspect, determining a corrected image frame in combination with one or more of the aspects from the 62nd to the 65th aspects includes determining a corrected image frame that has been warped to match the field of view of the first or second image sensor, based on a predicted disparity value.

[0140] In the 67th aspect, in combination with the 66th aspect, determining a corrected image frame includes determining a transformation matrix for the input image frame to warp the input image frame to the field of view of the other of the first or second image sensor.

[0141] In the 68th aspect, in combination with the 67th aspect, determining the transformation matrix includes determining the transformation matrix using computer vision processing (CVP).

[0142] In the 69th aspect, in combination with one or more of the 67th to 68th aspects, the 5 processor is further configured to perform a step including determining whether the confidence level associated with the transformation matrix is ​​below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the confidence level of the transformation matrix is ​​below a threshold level.

[0143] In the 70th aspect, the processor is further configured, either alone or in combination with one or more of the 62nd to 69th aspects, to perform a step including determining that the image characteristics of an input image frame are below a threshold level. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the image characteristics are below a threshold level.

[0144] In the 71st aspect, the processor 5 is further configured to perform a step including determining that the luminance of an input image frame is below a threshold level, either alone or in combination with one or more of the 62nd to 70th aspects. The step of determining a corrected image frame based at least in part on the input image frame and the predicted disparity value is performed based on the determination that the luminance is below a threshold level.

[0145] In the 72nd aspect, the processor is further configured, either alone or in combination with one or more of the 62nd to 71st aspects, to perform a step including determining a video sequence comprising a first image frame from a first image sensor, a second image frame from a second image sensor, and a corrected image frame. The corrected image frame appears in the video sequence between the first image frame and the second image frame.

[0146] In the 73rd aspect, the step of determining a corrected image frame, either alone or in combination with one or more of the 62nd to 72nd aspects, is further based on the image frame of the other of the first or second image sensor.

[0147] In the 74th aspect, either alone or in combination with one or more of the 62nd to 73rd aspects, the first disparity value represents the difference in the field of view along the first axis.

[0148] In the 75th aspect, in combination with the 74th aspect, the processor is configured to perform a step further comprising receiving a second disparity value indicating the difference in the field of view between a first image sensor and a second image sensor along a second axis different from the first axis. The step of determining the model is further at least in part based on the second disparity value.

[0149] In the 76th aspect, determining a model, either alone or in combination with one or more of the 62nd to 75th aspects, includes storing multiple disparity values. The model is based on the multiple disparity values.

[0150] In the 77th aspect, in combination with the 76th aspect, the previous value among the multiple disparity values ​​is replaced with a first disparity value based on at least one of the times associated with some of the multiple disparity values ​​or the previous value.

[0151] In the 78th aspect, the device is configured for a depth sensor, either alone or in combination with one or more of the 62nd to 77th aspects. A processor is coupled to the depth sensor and configured to receive a first depth value from the depth sensor.

[0152] In the 79th aspect, in combination with the 78th aspect, the depth sensor includes a range imaging system.

[0153] In the 80th aspect, in combination with the 79th aspect, the depth sensor includes a time-of-flight (ToF) system. Determining a first depth value includes determining a first depth value based on ToF measurements from the ToF system.

[0154] In the 81st aspect, in combination with one or more of the 79th to 80th aspects, the depth sensor includes a light detection and ranging (LIDAR) system. Determining a first depth value includes determining a first depth value based on LIDAR measurements from the LIDAR system.

[0155] In the 82nd aspect, either alone or in combination with one or more of the 62nd to 81st aspects, the apparatus is configured with respect to a computer vision processor (CVP) coupled to a processor, the CVP being configured to perform operations including receiving a first image frame from a first image sensor, receiving a second image frame from a second image sensor, and feature matching between the first image frame and the second image frame to determine the difference in the field of view between the first image sensor and the second image sensor. The processor is configured to receive the difference in the field of view between the first image sensor and the second image sensor from the CVP as a first disparity value.

[0156] Those skilled in the art will understand that information and signals can be represented using any of the various different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description can be represented by voltage, electric current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0157] With respect to Figures 1, 2, and 3, the components, functional blocks, and modules described herein include, among other examples, processors, electronic devices, hardware devices, electronic components, logic circuits, memory, software code, firmware code, or any combination thereof. In addition, the features described herein may be implemented via dedicated processor circuits, via executable instructions, or a combination thereof.

[0158] Those skilled in the art will further understand that the various exemplary logic blocks, modules, circuits, and algorithmic steps described herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly demonstrate this hardware-and-software compatibility, various exemplary components, blocks, modules, circuits, and steps have been outlined above in relation to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functions in various ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure. Those skilled in the art will also readily recognize that the order or combination of components, methods, or interactions described herein is merely illustrative, and that components, methods, or interactions of various aspects of this disclosure may be combined or performed in ways other than those illustrated and described herein.

[0159] The various exemplary logics, logic blocks, modules, circuits, and algorithmic processes described herein in relation to the implementation forms disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. Hardware and software compatibility is briefly described functionally and illustrated in the various exemplary components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0160] Hardware and data processing devices used to implement the various exemplary logics, logic blocks, modules, and circuits described in relation to the embodiments disclosed herein may be implemented or run using general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some implementations, the processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working with a DSP core, or any other such configuration. In some implementations, specific processes and methods may be performed by circuits specific to a given function.

[0161] In one or more embodiments, the functions described may be implemented in hardware, digital electronic circuits, computer software, firmware, or any combination thereof, including the structures disclosed herein and their structural equivalents. Implementations of the subject matter described herein may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device.

[0162] When implemented in software, the functionality may be stored on or transmitted via a computer-readable medium as one or more instructions or code. The processes of the methods or algorithms disclosed herein may be implemented in a processor-executable software module that resides on a computer-readable medium. Computer-readable medium includes both computer storage and communication media, including any medium that can enable the transfer of computer programs from one location to another. Storage media may be any available medium that can be accessed by a computer. Such computer-readable media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection may also be appropriately referred to as computer-readable medium. The terms "disk" and "disc" as used herein include compact discs (CDs), laser discs, optical discs, digital multipurpose discs (DVDs), floppy disks, and Blu-ray discs, where a disk typically reproduces data magnetically, and a disc reproduces data optically using a laser. Any combination of these should also be included within the scope of computer-readable media. In addition, the operation of a method or algorithm may reside on machine-readable and computer-readable media, as one or any combination or set of code and instructions, which can be incorporated into computer program products.

[0163] Various modifications of the implementations described herein may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to several other implementations without departing from the spirit or scope of this disclosure. Accordingly, the claims should not be limited to the implementations shown herein, but should be given the broadest scope consistent with this disclosure, the principles disclosed herein, and the novel features.

[0164] In addition, it will be readily understood by those skilled in the art that the terms “upper” and “lower” are sometimes used to simplify the description of a figure and indicate a relative position corresponding to the orientation of the figure on a properly oriented page, and may not reflect the proper orientation of any implemented device.

[0165] Some features described herein in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented separately or in any suitable partial combination in multiple implementations. Furthermore, features may be described above as working in several combinations, and may even be initially claimed as such, but one or more features from a claimed combination may, in some cases, be removed from that combination, and the claimed combination may be a partial combination or a variation of a partial combination.

[0166] Similarly, while actions are shown in a specific order in the diagrams, this should not be understood as requiring that such actions be performed in a specific or sequential order, or that all illustrated actions be performed, in order to achieve the desired result. Furthermore, the diagrams may schematically illustrate another exemplary process in the form of a flowchart. However, other actions not shown may be incorporated into the schematically illustrated exemplary process. For example, one or more additional actions may be performed before, after, simultaneously with, or between any of the illustrated actions. In some situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementation forms described above should not be understood as requiring such separation in all implementation forms, and it should be understood that the program components and systems described may generally be integrated together within a single software product or packaged within multiple software products. In addition, several other implementation forms fall within the scope of the following claims. In some cases, the actions embodied in the claims may be performed in a different order and still achieve the desired result.

[0167] As used herein, including in the claims, the term “or” means that when used in a list of two or more items, any one of the listed items may be taken alone, or any combination of two or more of the listed items may be taken. For example, if a composition is described as comprising components A, B, or C, the composition may include A only, B only, C only, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B and C. Also, as used herein, including in the claims, “or” when used in a list of items ending in “at least one of” indicates a disjunctive list, such as when the list “at least one of A, B, or C” means any of these in A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination thereof. The term “substantially” is defined as, as understood by those skilled in the art, as the majority of but not necessarily the entirety of what is specified (and including what is specified, for example, substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel). In any disclosed implementation, the term “substantially” may be replaced with “within [percentage] of” the specified percentage, including 0.1, 1, 5, or 10 percent.

[0168] The foregoing descriptions in this disclosure are provided so that any person skilled in the art can create or use this disclosure. Various modifications of this disclosure will be readily apparent to a person skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Accordingly, this disclosure should be given the broadest scope that is consistent with the principles and novel features disclosed herein, and is not limited to the examples and designs described herein. [Explanation of Symbols]

[0169] 100 devices 101 First image sensor, image sensor, sensor 102 Second image sensor, image sensor, sensor 104 Processors 106 memory 108 Computer Executable Instructions, Instructions 112 Image signal processors, ISPs 114 displays 116 Input / Output (I / O) Components, I / O Components 118 Power supply 131 First lens, lens 132 Second lens, lens 133 Autofocus (AF) system, AF system 134 Autofocus (AF) system, AF system 140 Depth Sensor 150 Sensor Hub 200 Systems 202 Computer Vision (CV) Processor (CVP), CV Processor 210 Parallax predictor, predictor 212 Model 214 Estimator 300 Systems 310 Geometric Warp Module, Warp Module 330 Corrected Image Frames 340 sequences 400 graphs 402 First line, line 404 Second line, line 412 data 414 data 500 ways 600 ways

Claims

1. A method performed by a computer for correcting an input image, The steps include receiving a first disparity value indicating the difference in the field of view between a first image sensor and a second image sensor, The steps include receiving a first depth value corresponding to the first disparity value, A step of determining a model configured to predict the disparity value between the field of view of the first image sensor and the field of view of the second image sensor at multiple depth values, wherein the model is at least partially based on the first disparity value and the first depth value. The steps include receiving the input image from either the first image sensor or the second image sensor, The steps include receiving a second depth value corresponding to the input image, The steps include determining a predicted disparity value based on the aforementioned model and the second depth value, The steps include determining a transformation matrix for warping the input image to the field of view of the other of the first or second image sensor, The steps include determining that the confidence level associated with the transformation matrix is ​​below a threshold level and that the predicted disparity value should be used, A step of determining a corrected image based on the input image and the predicted disparity value. A method comprising the step of determining the corrected image, wherein the step of determining the corrected image is determined as having been warped to match the field of view of the first image sensor or the other of the second image sensor based on the predicted disparity value.

2. The method according to claim 1, wherein the step of determining the transformation matrix includes the step of determining the transformation matrix using computer vision processing (CVP).

3. The step of determining that the image characteristics of the input image fall below a second threshold level. It further includes, The step of determining the corrected image is performed based on the determination that the image characteristics fall below the second threshold level. The method according to claim 1.

4. The step of determining that the brightness of the input image falls below a third threshold level. It further includes, The step of determining the corrected image is performed based on the determination that the brightness is below the third threshold level. The method according to claim 1.

5. A step of determining a video sequence including a first image from the first image sensor, a second image from the second image sensor, and the corrected image, wherein the corrected image appears in the video sequence between the first image and the second image. The method according to claim 1, further comprising:

6. The method according to claim 1, wherein the step of determining the corrected image is further based on the image of the other of the first or second image sensor.

7. The first parallax value indicates the difference in the field of view along the first axis. The method described above is A step of receiving a second disparity value indicating the difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, wherein the step of determining the model is further based at least in part on the second disparity value. The method according to claim 1, further comprising:

8. The step of determining the model is, A step of storing multiple disparity values, wherein the model is based on the multiple disparity values, A step of replacing the previous value among the plurality of disparity values ​​with the first disparity value based on at least one of several values ​​among the plurality of disparity values ​​or a time associated with the previous value. The method according to claim 1, including the method described in claim 1.

9. The method according to claim 1, wherein the first depth value includes the autofocus depth corresponding to the input image.

10. The process further includes the step of determining the first depth value based on range imaging, The method according to claim 1, wherein the step of determining the first depth value includes the step of determining the first depth value based on a time-of-flight (ToF) measurement or a light detection and ranging (LIDAR) measurement.

11. A device for correcting an input image, One or more processors, The device comprises one or more processors and a memory for storing instructions, and when an instruction is executed by one or more processors, Receiving a first disparity value indicating the difference in the field of view between the first image sensor and the second image sensor, Receiving a first depth value corresponding to the first disparity value, Determining a model configured to predict the disparity value between the field of view of the first image sensor and the field of view of the second image sensor at multiple depth values, wherein the model is determined to be at least partially based on the first disparity value and the first depth value. Receiving the input image from either the first image sensor or the second image sensor, Receiving a second depth value corresponding to the aforementioned input image, The predicted disparity value is determined based on the aforementioned model and the second depth value, Determining a transformation matrix for warping the input image to the field of view of the other of the first or second image sensor, It is determined that the confidence level associated with the transformation matrix is ​​below a threshold level, and therefore the predicted disparity value should be used. A corrected image is determined based on the input image and the predicted disparity value. A device that performs an operation including determining the corrected image, which includes determining the corrected image as warped to match the field of view of the first image sensor or the other of the second image sensors based on the predicted disparity value.

12. The device according to claim 11, further configured to perform the method described in any one of claims 2 to 10.

13. The first image sensor comprising a first field of view, and the second image sensor comprising a second field of view that at least partially overlaps with the first field of view, A depth sensor comprising one or more processors coupled to the depth sensor and configured to receive the first depth value from the depth sensor, A computer vision processor (CVP) coupled to one or more of the aforementioned processors, Receiving a first image from the first image sensor, Receiving a second image from the second image sensor, In order to determine the difference in field of view between the first image sensor and the second image sensor, feature matching is performed between the first image and the second image. Further comprising a CVP configured to perform operations including, The device according to claim 11, wherein one or more processors are configured to receive the difference in the field of view between the first image sensor and the second image sensor as a first disparity value from the CVP.

14. A non-temporary computer-readable medium for storing instructions, wherein when the instructions are executed by the processor of the device, the device causes the device to perform an operation including the method according to any one of claims 1 to 10.

Images