Combined electronic image stabilization and optical image stabilization

Abstract

Examples are described for combining optical image stabilization and electronic image stabilization to capture and process frames for a video. An example device configured to perform electronic image stabilization according to performed optical image stabilization can include a memory and one or more processors. The one or more processors can be configured to obtain optical image stabilization (OIS) information for OIS performed during capture of a series of frames by an image sensor, and determine an electronic image stabilization (EIS) filter based on the OIS information. The one or more processors can also be configured to obtain camera position information, and the EIS filter can also be based on the camera position information. The one or more processors can also configure an image signal processor to perform EIS based on the EIS filter.

Description

[0001] This application is a divisional application of the Chinese patent application No. 202080043936.6 (PCT / US2020 / 032630) with the title of “Combined Electronic Image Stabilization and Optical Image Stabilization” and a filing date of May 13, 2020. TECHNICAL FIELD

[0002] The present disclosure relates generally to image capture systems and devices, including combined electronic image stabilization and optical image stabilization. BACKGROUND

[0003] Many devices and systems include one or more cameras to capture images or video of a scene. Global motion, translational motion, or rotational motion of the device caused by, for example, a user’s hand causing the device to shake, can result in blurring of the images or video. Blurring of objects in the scene (or of the entire scene) is undesirable to the user. SUMMARY

[0004] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] Examples are described that combine optical image stabilization for capture of a series of frames with electronic image stabilization for post-capture processing of the series of frames for a video. An example device configured to perform electronic image stabilization in accordance with optical image stabilization can include a memory and one or more processors. The one or more processors can be configured to obtain optical image stabilization (OIS) information of OIS performed during capture of a series of frames by an image sensor, and determine an electronic image stabilization (EIS) filter based on the OIS information. The one or more processors can also be configured to obtain camera position information, and the EIS filter can be further based on the camera position information. The one or more processors can also configure an image signal processor to perform EIS based on the EIS filter.

[0006] In some implementations, the position of the movable camera lens is relative to the image sensor. The one or more processors can be configured to determine a lens distortion correction (LDC) transform based on the position of the movable camera lens during capture of the series of frames, and the EIS filter is based on the LDC transform. In this way, the device can also be capable of performing LDC in the case that OIS is performed. LDC can accurately remove lens distortion in the presence of OIS.

[0007] An example method for performing EIS can include obtaining OIS information for OIS performed during capture of a series of frames by an image sensor. The method also includes determining an EIS filter based on the OIS information. In some implementations, the method can include obtaining camera position information, and the EIS filter can be further based on the camera position information. In some implementations, the method can include configuring an image signal processor to perform EIS based on the EIS filter.

[0008] In some implementations, the position of the movable camera lens is relative to the image sensor. The method can include determining an LDC transform based on the position of the movable camera lens during capture of the series of frames, and the EIS filter is based on the LDC transform. In this way, LDC can be accurately performed while OIS is performed.

[0009] An example computer-readable medium can store instructions that, when executed by one or more processors of a device, cause the device to obtain OIS information for OIS performed during capture of a series of frames by an image sensor. Execution of the instructions also causes the device to determine an EIS filter based on the OIS information. In some implementations, execution of the instructions can cause the device to obtain camera position information, and the EIS filter can be further based on the camera position information.

[0010] In some implementations, the position of the movable camera lens is relative to the image sensor. Execution of the instructions can further cause the device to determine an LDC transform based on the position of the movable camera lens during capture of the series of frames, and the EIS filter is based on the LDC transform. In this way, the device can accurately perform LDC while also performing OIS.

[0011] An example device for performing EIS can include means for obtaining OIS information for OIS performed during capture of a series of frames by an image sensor. The device also includes means for determining an EIS filter based on the OIS information. In some implementations, the device can include means for obtaining camera position information, and the EIS filter can be further based on the camera position information.

[0012] The device can also include a unit for obtaining camera position information, and the EIS filter is further based on the camera position information. The EIS filter can include a shift function that indicates, for each of a plurality of pixel positions in a series of pre-processed frames, a shift of the pixel position in the un-processed frame to an associated pixel position in the processed frame. The device can also include a unit for determining a LDC transform based on a position of a movable camera lens relative to an image sensor during capture of the series of frames. The EIS filter is based on the LDC transform. In this way, LDC can be accurately performed in the case that OIS is performed.

[0013] The device can include a unit for determining a rotational transform for reducing rolling shutter distortion. The rotational transform is based on a difference between an angle of the image sensor at readout of each image sensor pixel and an angle of the image sensor at readout of a center image sensor pixel. The EIS filter is based on the rotational transform. BRIEF DESCRIPTION OF DRAWINGS

[0014] Aspects of the disclosure are illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which like references indicate similar elements.

[0015] Figure 1 A flow diagram showing different distortions added during frame capture.

[0016] Figure 2 A depiction of an example optical image stabilization system including a shift lens is shown.

[0017] Figure 3 A depiction of an example light guide based on optical image stabilization using a shift lens is shown.

[0018] Figure 4 A depiction of an example optical image stabilization system including rotation of a lens and image sensor is shown.

[0019] Figure 5 A flow diagram showing different distortions added during frame capture when performing optical image stabilization is shown.

[0020] Figure 6 A block diagram of an example device configured to perform optical image stabilization and electronic image stabilization is shown.

[0021] Figure 7 A block diagram of an example system for configuring electronic image stabilization based on performed optical image stabilization is shown.

[0022] Figure 8 An illustrative flow diagram depicting example operations for performing electronic image stabilization based on performed optical image stabilization is shown.

[0023] Figure 9 An illustrative flow diagram depicting example operations for calibrating measurements from OIS to be used for EIS is shown. DETAILED DESCRIPTION

[0024] Aspects of the disclosure can be used for image stabilization of image or video capture. In some example implementations, electronic image stabilization is combined with optical image stabilization when processing a sequence of frame captures.

[0025] Many cameras, such as those included in handheld devices, including smartphones, tablets, digital cameras, and the like, can have global movement during video capture. For example, a user’s hand can shake, or the user can rotate the camera to capture different directions. The camera movement or motion can include one or more of translational motion, such as motion along one or more axes without rotation, or rotational motion, such as one or more of roll, pitch, or yaw. The camera captures a series of frames to capture a video, and movement of the camera during capture can cause the scene to jump between frame captures. For example, the scene can move based on translational motion, and the scene can rotate based on rotational motion. If the frame rate of the camera is not fast enough, the scene in the video can appear blurry due to movement of the camera.

[0026] Additionally, the camera can include a lens that focuses light onto the image sensor. The curvature of the lens or lens defects can cause light in the scene to refract to the image sensor, causing the scene to appear distorted in the video. For example, the intensity of light at the corners of the image sensor can be less than the intensity of light at the center of the image sensor due to the lens refracting light towards the center of the image sensor. In another example, a portion of the scene captured through the edges of the image sensor can appear warped due to additional curvature of the edges of the lens relative to the center of the lens, such as a fisheye effect in the frame.

[0027] Another distortion can be attributed to the rolling shutter of the camera. If the camera has a global shutter, the entire pixels of the image sensor are read out at once. In this way, each pixel captures the scene at the same time. However, many cameras have a rolling shutter. With a rolling shutter, only a portion of the pixels of the image sensor are read out at a time. Thus, multiple readouts occur in succession to capture a frame. For example, the readout of the pixels can snake from the top left pixel of the image sensor to the bottom right pixel of the image sensor. In this way, the top left pixel captures the scene at an earlier time than the bottom right pixel captures the scene. The different times in the capture due to the rolling shutter can cause motion blur in the frame.

[0028] Figure 1A flowchart 100 showing different distortions added during frame capture. A scene 102 is to be captured in a frame by a camera. If the camera has a camera movement 104, then the scene 102 is a motion-blurred scene 106 due to the camera movement 104. Additionally, the motion-blurred scene 106 has added lens distortion 108 to result in a lens-distorted, motion-blurred scene 110. If the camera has a rolling shutter 112, what the camera's image sensor captures is a rolling-shutter-distorted, lens-distorted, motion-blurred scene 114.

[0029] Electronic image stabilization (EIS) is a post-capture processing of frames to reduce one or more of the distortions during capture. For example, EIS can compensate for global motion of a camera to reduce jitter in a video. EIS can also be used to reduce distortions caused by a rolling shutter. As used herein, EIS can also involve a reduction in lens distortion (such as transforming or mapping pixels in a captured frame to real locations in a scene in order to de-warp the captured frame). In some examples of EIS that reduces global motion, the size of each final image in a final stream of images of a video can be smaller than the size of the frames captured by a camera sensor. For example, a device can crop a portion (such as 10%) of each of the captured frames (the crop location moves based on the global motion) to generate a final, processed stream of images that is a fraction of the pixel size of the frames captured by the camera sensor. As described above, the crop window can be shifted to compensate for camera motion. For example, if the camera moves from left to right, the crop window can move from right to left in the captured frames. The frames can also be rotated, stretched, or otherwise adjusted to compensate for rotational motion.

[0030] Because EIS is a post-capture process, EIS can not be able to compensate for some motion blur. For example, if the blur in a frame is caused by motion and the frame rate of an image sensor, then EIS can not be able to sufficiently compensate for the motion blur. Such motion blur can also be added as the exposure window of a frame increases (e.g., when the International Organization for Standardization (ISO) setting of a camera is reduced from 800 to 100).

[0031] Another method to reduce one or more of the distortions during capture is optical image stabilization (OIS). In some implementations of OIS, one or more components of a camera can be moved to compensate for camera motion. For example, a device can move a camera lens to counteract hand jitter of a user during video capture. The camera can be coupled to a motion sensor (such as a gyroscope) to indicate motion of the camera during image capture, and the camera can adjust the camera lens position based on the gyroscope's measurements.

[0032] Types of OIS include lens shift OIS and module tilt OIS. Figure 2A depiction of an example lens shift OIS system 200 for a camera is shown. The system can be included in a camera housing 202, and the system can include a shiftable lens 204 (which can be moved as indicated by movement 212 relative to an image sensor 206). While movement 212 is depicted as being along an axis, movement 212 can be along a plane in three-dimensional space (i.e., in two perpendicular directions). Figure 2 While movement 212 is depicted as being along an axis, movement 212 can be along a plane in three-dimensional space (i.e., in two perpendicular directions).

[0033] In some implementations, system 200 uses one or more magnets (such as magnets 210A and 210B) to shift lens 204. While magnets 210A and 210B are shown, other units for shifting lens 204 can be used, such as mechanical units (e.g., push or pull mechanisms) or electrical units (e.g., based on electrical charges to shift lens 204). System 200 can include a lens position sensor 208 to determine the lens position relative to image sensor 206. In some implementations, lens position sensor 208 can be a Hall effect sensor (also referred to as a Hall sensor) to indicate the position of lens 204. For example, the lens position can be based on the magnetic field strength caused by magnet 210A or 210B (or other magnets). The Hall sensor measures the magnetic field strength and outputs a voltage corresponding to the magnetic field strength. In this way, the Hall sensor measurements can be compared to the gyroscope measurements (after conversion to a common measurement system) to determine whether lens 204 is to be shifted. System 200 can then control the magnets to shift lens 204.

[0034] In some implementations, lens 204 includes a concave lens to direct light to different portions of image sensor 206 based on the lens position. Figure 3 A depiction 300 of example light direction based on lens shift OIS is shown. The example OIS system includes a fixed lens 304 (relative to an image sensor 302) and an adjustable shiftable lens 306 (which can be referred to as a shiftable lens or an adjustable lens). The curvature of adjustable lens 306 can be concave (as shown) to direct light toward image sensor 302. In the illustration, image sensor 302 is initially turned 90 degrees relative to the light path (original light path 308) from a captured object, and adjustable lens 306 is in a middle or center position (indicated by the dashed outline). As a result, light from the object along original light path 308 is directed to camera sensor position 316 by fixed lens 304 and adjustable lens 306. When the camera moves, the light from the object follows a new light path 310. If adjustable lens 306 is not shifted, the light along new light path 310 can be directed to position 318 of image sensor 302.

[0035] However, the camera sensor 302 will appear in the captured frame to be held at 90 degrees relative to the object. Thus, light along the new light path 310 will reach the image sensor 302 at location 316 (same as the original light path 308). To compensate for the camera motion, the system 300 can adjust the position of the adjustable lens 306 (as indicated by lens movement 312), and the combination of lens curvature of the fixed lens 304 and the adjustable lens 306 directs light from the new light path 310 along the adjusted light path 314 to the location 316 of the image sensor 302.

[0036] The mapping between the gyroscope measurements and the adjustable lens position can be used to determine the direction and magnitude of the movement 312. In determining the magnitude and direction of the lens movement 312, the gyroscope can be sampled to determine the change in position and orientation of the image sensor 302, and the mapping is used to convert the change in position and orientation to the movement 312.

[0037] Another type of OIS is a module tilt OIS (which can also be referred to as a rotation-based OIS or a rotational OIS). Figure 4 A depiction of an example module tilt OIS system 400 is shown. The system 400 can be included in a camera housing 402. The system 400 includes a lens 404 with a fixed position relative to an image sensor 406, and the lens 404 and the image sensor 406 can be tilted or rotated (as indicated by movement 412). In some implementations, the system 400 can use an actuator or other rotational unit to rotate the lens 404 and the image sensor 406. Similar to the lens shift method, the lens 404 and the image sensor 406 can be rotated to keep light from an object that is fixed at a position on the image sensor 406 during camera movement.

[0038] The system 400 can include a lens position sensor 408 to determine the lens position, such as the angle of rotation. In some implementations, the lens position sensor 408 is a photodetector, and the light intensity to the photodetector measured by the photodetector indicates the lens position. For example, if the lens position is as shown by the dashed box, the photodetector is covered, and no light reaches the photodetector. If the lens position is at the opposite end (as shown), the photodetector is not covered, and the light intensity reaches a maximum at the photodetector. Thus, the center lens position is associated with a center light intensity measurement. In another example, the sensor can measure the rotation at the actuator 410 to determine the lens position. Any other suitable unit for determining the lens position can also be used.

[0039] In this way, the lens position sensor measurements can be compared (after conversion to a common measurement system) with the gyroscope measurements to determine whether the lens 404 is to be shifted (by rotating the lens 404 and image sensor 406). The system 400 can then control the actuator 410 to rotate the lens 404 and image sensor 406.

[0040] In some implementations, the sampling rate of the gyroscope can be 6 kilohertz (kHz), and the lens 204( Figure 2 ) can be configured to shift, or the lens 404( Figure 4 ) can be configured to rotate at the same rate. In this way, for a camera that captures video at, for example, 24 to 120 frames per second (fps), the camera lens 204 or 404 can be adjusted tens to hundreds of times for each frame. In this way, each frame can be stabilized to reduce blur within the frame.

[0041] While OIS can sometimes reduce camera blur better than EIS, the range of camera lens movement for OIS is limited by design and physical constraints. For example, the limitations on lens movement can limit OIS to compensating for up to about one degree of camera movement. In contrast, EIS can compensate for up to five degrees or more of camera movement. However, a camera is limited to performing either OIS or EIS to achieve image stabilization.

[0042] In some implementations, OIS is combined with EIS to improve image stabilization. While OIS can reduce one or more distortions caused by camera movement, OIS can introduce different factors or distortions for EIS. In some implementations, the lens position during OIS is used to determine EIS during post-capture processing.

[0043] Figure 5 A flowchart 500 is shown that illustrates different distortions added during frame capture when OIS is performed. Similar to Figure 1, the scene 502 can be affected by camera movement 504 to produce a motion blurred scene 506. OIS can move the lens to reduce the motion blur (indicated by OIS-based lens displacement 508), but the lens displacement can cause a change in lens distortion. Thus, the lens-displaced, reduced-motion-blur scene 510 can have a changed lens distortion 512 (based on the change in lens position) to result in a lens-displaced, reduced-motion-blur scene 514 based on the change in lens distortion. The rolling shutter 516 can cause distortion (which can be reduced by OIS), but there can still be. The OIS lens displacement can also affect the distortion caused by the rolling shutter. Thus, the camera's image sensor can capture a rolling-shutter-distorted, lens-displaced, reduced-motion-blur scene 518. As noted above, the lens displacement of OIS can be faster than frame capture. In some implementations, EIS is configured to consider multiple lens positions for each captured frame. When combining OIS and EIS for video capture, only one of OIS or EIS is referenced to improve image stabilization.

[0044] In the following description, numerous specific details are set forth such as examples of specific components, circuits, and processes to provide a thorough understanding of the present disclosure. The term "coupled" as used herein means connected directly to or through one or more intervening components or circuits. Also, in the following description and for purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that these specific details can not be required to practice the teachings presented herein. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the teachings of the present disclosure. Some portions of the detailed description that follows are presented in terms of procedures, logic blocks, processing and other symbolic representations of operations on data bits within computer memories. In the present disclosure, procedures, logic blocks, processing and the like are conceived to be self-consistent sequences of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0045] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the present application, discussions utilizing terms such as "accessing," "receiving," "sending," "using," "selecting," "determining," "normalizing," "multiplying," "averaging," "monitoring," "comparing," "applying," "updating," "measuring," "deriving," "resolving," or the like, refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0046] In the drawings, a single block can be described as performing one or more functions; however, in actual practice, the one or more functions performed by the block can be performed in a single component or across multiple components, and / or can be performed using hardware, using software, or using a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. In addition, the example devices can include components other than those shown, including well-known components such as a processor, memory, etc.

[0047] Aspects of the present disclosure are applicable to any suitable electronic device capable of capturing video, such as a security system, a smartphone, a tablet, a notebook computer, a digital video and / or still camera, a web camera, a VR headset, an AR headset, etc., and one or more cameras. Although described below with respect to a device having or coupled to one camera, aspects of the present disclosure are applicable to devices having any number of cameras, and thus are not limited to devices having one camera.

[0048] The term "device" is not limited to one or a particular number of physical objects (such as one smartphone, one camera controller, one processing system, etc.). As used herein, a device can be any electronic device having one or more portions that can implement at least some portions of the present disclosure. Although the description and examples below use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a particular configuration, type, or number of objects.

[0049] Figure 6A block diagram showing an example device 600 for performing OIS and EIS. The example device 600 can include or be coupled to a camera 601, a gyroscope 620, a processor 604, a memory 606 storing instructions 608, a camera controller 610, and a sensor controller 622. The device 600 can optionally include (or be coupled to) a display 614 and a plurality of input / output (I / O) components 616. The device 600 can include additional features or components not shown. For example, a wireless interface for a wireless communication device can be included, which can include a plurality of transceivers and baseband processors. The device 600 can include or be coupled to additional cameras other than the camera 601 and additional sensors other than the gyroscope 620.

[0050] The camera 601 can be capable of capturing a series of image frames (which can be referred to as frames or images) for video capture. The camera 601 includes an image sensor and at least one camera lens 602 associated with the image sensor. The camera 601 can be configured for OIS, and the position of the lens 602 can be adjustable (such as by shifting or rotating the lens 602).

[0051] The gyroscope 620 can be capable of providing information about the position of the camera 601 (such as position due to translational motion and orientation due to rotational motion). As used herein, position information can account for both translational motion and rotational motion of the camera 601. In some implementations, the device 600 can be configured to sample the gyroscope 620 (such as via the sensor controller 622) to obtain position information of the camera 601 during frame capture by the camera 601. In this way, the device 600 is configured to use gyroscope measurements to determine lens movement for OIS. For example, the camera controller 610 can provide instructions to the camera 601 for controlling OIS during video capture. Although not shown, the sensor controller 622 can be directly coupled to the camera controller 610, or the gyroscope measurements can be provided to the camera controller in another manner different from that shown for controlling OIS. Additional or alternative motion sensors can be included or coupled to the device 600, such as an accelerometer, a magnetometer, etc. In some implementations, the sensor controller 622 is a sensor hub for receiving measurements from a plurality of sensors.

[0052] Device 600 can be configured to perform EIS on a series of frames after capture. In some implementations, image signal processor 612 is configured to receive the captured series of frames and perform EIS. In this way, image signal processor 612 can configure the EIS to be performed (e.g., adjusting one or more EIS filters) based on gyroscope measurements received by camera controller 610. Image signal processor 612 can also configure the EIS to be performed based on the position of lens 602 during the capture of the series of frames.

[0053] The memory 606 may be a non-transient or non-temporary computer-readable medium storing computer-executable instructions 608 to perform all or part of one or more operations described in this disclosure (e.g., for performing OIS or EIS). The device 600 may also include a power supply 618, which may be coupled to or integrated into the device 600.

[0054] Processor 604 may be one or more suitable processors capable of executing scripts or instructions of one or more software programs (such as instructions 608) stored in memory 606. For example, processor 604 may perform operations for OIS or EIS (such as determining the configuration for image signal processor 612 to perform EIS). In some aspects, processor 604 may be one or more general-purpose processors that execute instructions 608 to cause device 600 to perform any number of functions or operations. For example, processor 604 may be an application processor for device 600 (such as for a smartphone or other computing device). In other or alternative aspects, processor 604 may include integrated circuits or other hardware to perform functions or operations without using software. In some implementations, camera controller 610 may be configured to perform OIS or EIS operations, which may include executing instructions 608 or instructions locally stored in camera controller 610.

[0055] Despite Figure 6 The example is shown as coupled to each other via processor 604, but processor 604, memory 606, camera controller 610, sensor controller 622, optional display 614, and optional I / O components 616 can be coupled to each other in various arrangements. For example, processor 604, memory 606, camera controller 610, sensor controller 622, optional display 614, and / or optional I / O components 616 can be coupled to each other via one or more local buses (not shown for simplicity).

[0056] The display 614 can be any suitable display or screen that allows user interaction and / or presents items, such as captured images or video, for viewing by a user. In some aspects, the display 614 can be a touch-sensitive display. The I / O components 616 can be or include any suitable mechanism, interface, or device that receives input from a user, such as commands, and provides output to the user. For example, the I / O components 616 can include, but are not limited to, a graphical user interface, a keyboard, a mouse, a microphone, and a speaker, among others.

[0057] The camera controller 610 can include an image signal processor 612, which can be one or more image signal processors to process captured image frames captured by the camera 601. The camera controller 610, such as the image signal processor 612, can perform operations associated with OIS and / or EIS, as well as process or generate processed image frames for video. The camera controller 610 can also include an encoder to encode the processed video. In some other implementations, the encoder can be executed by the processor 604 to encode the video. The encoded video can be in a format for playback or storage by the device 600 or another suitable device. As described above, the camera controller 610 can also control the operation of the camera 601. In some implementations, the camera controller 610, such as the image signal processor 612, can adjust or instruct the camera to adjust one or more camera components for OIS, such as adjusting the camera lens 602. For example, the camera controller 610 can obtain measurements from a lens displacement sensor of the camera 601, obtain measurements from the gyroscope 620, and provide specific instructions to one or more components of the camera 601 to move the lens 602. In some other implementations, the camera 601 is configured to receive the gyroscope measurements and move the lens 602 during OIS.

[0058] In some aspects, the image signal processor 612 can execute instructions from memory, such as instructions 608 from the memory 606 or instructions stored in a separate memory coupled to the image signal processor 612, to process frames captured by the camera 601. In some other aspects, the image signal processor 612 can include specific hardware. The image signal processor 612 can alternatively include a combination of specific hardware and the ability to execute software instructions.

[0059] The sensor controller 622 can be configured to sample the gyroscope 620 (and other sensors coupled to the sensor controller 622). In some aspects, the sensor controller 622 can include one or more processors to execute instructions from memory, such as instructions 608 from the memory 606 or instructions stored in a separate memory coupled to or included in the sensor controller 622. In some other aspects, the sensor controller 622 can include specific hardware. The sensor controller 622 can alternatively include a combination of specific hardware and the ability to execute software instructions. In some implementations, portions of the sensor controller 622 can be embodied in other components of the device 600, such as the processor 604, the camera controller 610, or the camera 601.

[0060] The device 600 is configured to perform OIS during frame capture to compensate for at least a portion of camera motion, and the device 600 is configured to perform EIS after frame capture to compensate for camera motion and other distortions, such as lens distortion. As described herein, OIS can cause respective distortions or otherwise affect EIS. For example, if OIS causes the lens 602 to move during frame capture, the positioning of lens distortion relative to the image sensor (and the captured frame) changes. In combining EIS with OIS, the device 600 is configured to account for the change in camera lens position used to perform EIS.

[0061] Figure 7 A block diagram of an example system 700 for configuring EIS based on OIS performed during frame capture is shown. Portions of the example system 700 can be embodied in the camera controller 610, the sensor controller 622, the processor 604, the memory 606, the camera 601, or any other suitable component of the device 600 in Figure 6 Each of the blocks in the system 700 can be embodied in hardware, software, or a combination of the two. As such, the block diagrams in Figure 7 the disclosure is not limited to the configuration of the example system 700 shown.

[0062] The gyroscope 720 can be an implementation of the gyroscope 620 in Figure 6 The sensor controller 722 can be an implementation of the sensor controller 622 in Figure 6implementation of the sensor controller 622 in FIG. 6. The sensor controller 722 can sample the gyroscope 720 (e.g., at a sampling rate of 6 kHz or another suitable rate), and the sampling can occur during frame capture by the image sensor 702. During frame capture, the system 700 is configured to perform OIS. During OIS, the OIS driver 716 can communicate with the OIS controller 708 to control movement of the lens 704 via the actuator 710. The OIS driver 716 can be implemented in hardware, software, or a combination of both, and can be implemented in the camera 601, the camera controller 610, or another suitable component of the device 600. Figure 6 implementation of the camera controller 610, the processor 604, or another suitable component of the device 600 in FIG. 6. The OIS controller 708 can be implemented in hardware, software, or a combination of both, and can be implemented in the camera 601, the camera controller 610, or another suitable component of the device 600. The movement of the lens 704 can be based on lens shift OIS or module tilt OIS. For module tilt OIS, the actuator 710 can be a Figure 4 example implementation of the actuator 410 in FIG. 4. While the actuator 710 is shown for controlling movement of the lens 704, any suitable component can be used. For example, the OIS system can use one or more magnets (such as the magnets 210A or 210B in FIG. 2), a current generator, or other unit that physically moves the lens 704. The motion 714 indicates example motion that the lens 704 can perform during OIS. While the motion 714 shown can be similar to motion relative to the image sensor 702 for lens shift OIS, any suitable lens motion can be performed when performing OIS (including rotating both the lens 704 and the image sensor 702). Figure 2

[0063] The OIS controller 708 obtains lens position measurements from the lens position sensor 706. In some examples, the lens position sensor 706 includes a Hall effect sensor or a photodetector. For example, for lens shift OIS that uses a magnet to move the lens 704, the OIS controller 708 can obtain a voltage (measuring a magnetic field) from the Hall effect sensor to determine the lens position (or can obtain a voltage representation that is converted, such as to a digital format). The OIS controller 708 provides the measurements to the OIS driver 716, and the OIS driver 716 provides an indication of the lens position to the native code set (NCS) module 718. The OIS controller 708 and the OIS driver 716 can also use values from the gyroscope 720 or values from the image sensor 702 to determine and control movement of the lens 704 via the actuator 710 during OIS.

[0064] ​The NCS module 718 is configured to convert different measurements to a common unit or units recognized by different modules. The NCS module 718 can be implemented in the camera controller 610, the processor 604, or another suitable component of the device 600, and can be implemented in hardware, software, or a combination of both. In some implementations, the NCS module 718 can convert gyro measurements to lens position measurements. For example, the NCS module 718 can convert gyro measurements to a digital representation of a voltage that would be provided by a Hall effect sensor. The NCS module 718 can provide the representation to the OIS driver 716, and the OIS driver 716 compares the representation from the NCS module 718 to the voltage indication from the OIS controller 708 to determine a value to provide to the OIS controller 708 to control the actuator 710. In another example, the NSC module 718 converts gyro measurements from the sensor controller 722 to a value in a common unit, and converts lens position sensor measurements from the OIS driver 716 to a value in the common unit so that the values can be compared to each other. In some implementations, the NCS module 718 includes a lookup table, script, or other suitable unit to map various values from different modules to values of other modules or to a common value. Here, the common unit can be referred to as a native code set. The NCS module 718 provides the NCS (including the converted values of the lens position measurements and the converted values of the gyro measurements) to the motion analysis module 734.

[0065] The motion analysis module 734 can be configured to compare the camera motion indicated by the gyro 720 and the lens motion indicated by the lens position sensor 706. The motion analysis module 734 can then reference the frame metadata 726 from the image sensor 702 to provide a result of the comparison between the lens position sensor measurements and the gyro measurements to the EIS filter module 736. The EIS filter module 736 can determine an EIS to perform based on the unprocessed, captured frame 740 from the image sensor 702 and the comparison result from the motion analysis module 734.

[0066] The motion analysis module 734 can be included in an EIS library for configuring the EIS transform 738. The EIS library can be implemented in the camera controller 610, the processor 604, or another suitable component of the device 600, and can be implemented in hardware, software, or a combination of both. The EIS library can include a lookup table, mapping, or other calculation of values (indicating comparisons) to pass to the EIS filter module 736 to configure the EIS transform 738. Example calculations or mappings are described below with reference to example methods for determining an EIS. The EIS filter module 736 determines an EIS by generating the EIS transform 738 based on the captured frame 740 and the values from the motion analysis module 734.

[0067] Based on the EIS transform 738, the image signal processor 714 is configured to perform EIS on the frame 724. In some implementations, the frame 724 is the same as the frame 740. In some implementations, the frame 740 can be a previous instance of the frame 724 (e.g., to determine parameters for EIS to evaluate EIS before using on the current frame 724). In some implementations, the frame 740 can be obtained from memory or a buffer.

[0068] One or more filters or EIS-specific components of the image signal processor 714 can be configured using the EIS transform 738. The image signal processor 714 processes the frame 724, which includes performing EIS. Other processing can include de-noising, edge enhancement, color balancing, or other suitable image processing filters. The image signal processor 714 can be an example implementation of the image signal processor 612 of the device 600 in Figure 6 In some other implementations, another suitable component of the device 600 can be configured to perform EIS.

[0069] The video encoder 730 encodes the processed frame 728 to generate an encoded video 732. The encoded video 732 can be provided for viewing, storage, or transmission (e.g., to another device). The video encoder 730 can be implemented in the image signal processor 612, another portion of the camera controller 610, the processor 604 (such as in instructions 608 to be executed by the processor 604 or another suitable processor), or another suitable component of the device 600. The video encoder 730 can be implemented in hardware, software, or a combination of both.

[0070] Figure 8 An illustrative flowchart depicting example operations 800 for performing EIS based on OIS is shown. Referring back to Figure 7 When EIS is determined based on performed OIS, the EIS transform 738 supplements the performed OIS to provide the desired combined image stabilization processing of the frame 724. Figure 8 The example operations 800 in Figure 6 may be performed by the device 600 in Figure 7 The system 700 in While described with reference to examples performed by the device 600, any suitable device or system can be used to perform the example operations 800 (or other example operations).

[0071] At 802, the device 600 obtains OIS information. Referring back to Figure 6, the camera controller 610 (or the processor 604) can obtain an indication of lens position sensor measurements regarding movement or position of the lens 602. For example, the lens position sensor measurements (e.g., Hall sensor measurements) can be converted to a distance (such as a pixel distance). The distance can then be converted to a value in the NCS. The OIS information can include the distance value in the NCS (or otherwise an indication of the lens position based on the lens position sensor measurements). Referring to Figure 7 , the motion analysis module 734 can obtain the NCS of the lens position measurements.

[0072] In some implementations, the device 600 can also obtain camera position information (804). Referring to Figure 6 , the camera controller 610 (or the processor 604) can obtain gyroscope measurements from the sensor controller 622. Referring to Figure 7 , the motion analysis module 734 can obtain the NCS of the gyroscope measurements. In some implementations, the camera position information can include an indication of the camera position based on the gyroscope measurements. For example, the gyroscope measurements can be converted to a rotation of the camera 601 (which can be represented by one or more Euler angles or quaternions). The measurements can also indicate a position of the camera (such as based on translational motion). The determined rotation of the camera 601 (e.g., Euler angles or quaternions) can be converted to a camera position value in the NCS. The camera position information can include the camera position value in the NCS.

[0073] In some implementations, the device 600 can obtain frame capture information (806). Referring to Figure 6 , the camera 601 captures a series of frames for capturing a video, and the camera controller 610 (or the processor 604) obtains metadata from the frames. For example, the frames can be in the Exchangeable Image File Format (EXIF), and each EXIF header (or another suitable location in each frame) can include metadata about the frame. The metadata can include a color palette, a color scheme (such as RGB or YuV), a light intensity, or other frame information used to render the frame as an image (or other frame information that can otherwise affect EIS).

[0074] At 808, the device 600 determines an EIS filter based on the OIS information. Referring to Figure 6 , the camera controller 610 (or the processor 604) can determine a configuration of the image signal processor 612 to perform EIS based on the performed OIS (indicated by the OIS information). Referring to Figure 7 , the EIS filter module 736 can determine an EIS transform 738 based on the NCS of the lens position measurements (such as based on processing of the NCS by the motion analysis module 734) to configure the image signal processor 714 to perform EIS.

[0075] In one example, the OIS information indicates a lens position of the camera 601. If the device 600 performs lens shift OIS, the lens position can be used to determine a change in lens distortion. For example, when the lens 602 is in a middle position (such as centered for the image sensor), a transform for a frame captured from the image sensor that is centered can be shifted based on a shift of the lens 602 during OIS. In this way, the device 600 can determine a shift of the transform based on the shift of the lens 602. The device 600 can also determine an effect of lens shift during readout of a rolling shutter on frame capture, and the device 600 can configure the image signal processor 612 to compensate for the moving lens 602 when reversing (or otherwise reducing) rolling shutter blur during EIS.

[0076] Returning to Figure 8 In some implementations of 808, the device 600 can further determine an EIS filter based on the camera position information (810). For example, the gyroscope measurements can indicate that the camera is moving more than the range that OIS can compensate for to keep the scene stable across a series of frames, as compared to the lens position measurements. In this way, the device 600 determines additional compensation for EIS to perform to reduce movement of the scene across a series of frames. See Figure 6 , the camera controller 610 (or the processor 604) can determine a configuration of the image signal processor 612 to perform EIS based on a comparison of the gyroscope measurements and the lens position measurements (from the frame metadata). See Figure 7 , the EIS filter module 736 can determine an EIS transform 738 to configure the image signal processor 714 to perform EIS based on the NCS including the lens position measurements and the gyroscope measurements (such as based on a comparison of the gyroscope measurements and the lens position measurements of the motion analysis module 734). Returning to Figure 8 In some other implementations of 808, the device 600 can further determine an EIS filter based on frame capture information (812). For example, the processing of the NCS (which can include an indication of the lens position or the camera position) can be based on frame capture information (such as the frame metadata 726). As such, the EIS transform 738 generated by the EIS filter module 736 Figure 7 ) can be based on the frame capture information.

[0077] After determining the EIS filter, the device 600 can perform EIS on the captured frame. For example, the image signal processor 612 performs EIS based on the OIS of the frame from the camera 601 to reduce distortion during capture. As described above, the two types of OIS include lens shift OIS and module tilt OIS. The effect of OIS on distortion during capture differs based on the type of OIS. For example, returning reference to Figure 5 Lens distortion 512 can be more affected by lens shift OIS than module tilt OIS due to the movement of the lens relative to the image sensor. In another example, returning reference to Figure 5 Module tilt OIS has a greater effect on rolling shutter distortion than lens shift OIS. Thus, the distortion to be reduced by EIS can be determined differently based on the type of OIS performed. Described herein are example implementations for determining the distortion to be reduced by EIS. In some implementations, a profile of the distortion is determined, and the image signal processor 612 is configured to perform EIS inverse to the profile of the distortion. Examples are described with reference to lens shift OIS and module tilt OIS.

[0078] Lens shift OIS

[0079] For lens shift OIS, the camera lens moves relative to the image sensor. Returning reference to Figure 7 , the lens position sensor 706 indicates the position of the lens 704. In some implementations, the OIS controller 708 or the OIS driver 716 converts the lens position sensor measurement from one unit to a measurement of the lens position relative to the image sensor 702. For example, the OIS controller 708 or the OIS driver 716 can determine the center of the lens 704 at pixel position (x, y) of the image sensor 702 based on the lens position sensor measurement. x (in X) can refer to the xth row of the image sensor 702, and y (in Y) can refer to the yth column of the image sensor 702 of size x rows by y columns. If the lens position sensor 706 is a Hall sensor that measures the magnetic field of a magnet used to position the lens 704, the OIS controller 708 can convert the voltage generated by the Hall sensor to a digital representation. The OIS driver 716 can convert the digital representation to a pixel position (x, y) of the image sensor 702. In some other implementations, the NCS module 718 can convert the digital representation to an NCS that indicates the pixel position (x, y) of the image sensor 702. In some implementations, the pixel position can be indicated as a shift from the center of the image sensor 702 (measured in pixels).

[0080] An example conversion of the lens position (as indicated by the Hall sensor) to a pixel shift is shown in equation (1) below:

[0081] OIS shift = a + b*OISLensPos + c*OISLensPos 2 + d*OISLensPos 3 (1)

[0082] OISLensPos is the lens position as indicated by the Hall sensor. OIS shift is the shift in units of pixels that converts the lens position. a, b, c, and d are parameters to be estimated for best converting the Hall sensor measurements to OIS shift . For example, the parameters a, b, c, and d can be determined during calibration of the OIS system and device, and the parameters can be stored for use in converting during device operation. For example, the parameter "a" can represent a fixed offset of the Hall sensor measurement from a zero shift value when the camera lens is at the middle position. The parameters "b" - "d" can map a finite impulse response for the conversion. When the conversion is shown as parameter "d", any appropriate number of parameters can be used, and any appropriate impulse response function can be used (such as including e*OISLensPos 4 , etc.).

[0083] OIS shift may be used for lens positions measured along a single axis. As such, if the lens is moved along a plane of lens shift OIS, there can be two OIS shift instances (one instance of motion defined for each axis). In some implementations, OIS shift may be the product of two third order polynomial functions (such as the product of two instances of equation (1)). The calibration examples as described herein provide examples of determining the parameters of the combined OIS shift function. To clearly describe aspects of the disclosure, examples of determining EIS use shift example functionality of Figure 1 . In some implementations, the combined OIS shift may be used, and examples are not limited to the shift function shown in equation (1).

[0084] In some implementations, the conversion can be based on a conversion of the camera's focal length and a particular Hall sensor reading to distance units. For example, from the perspective of the image sensor, the lens 602 is closer to the camera 601( Figure 6The image sensor of (1) makes lens shift look larger. In another example, various Hall sensors can be used for lens shift OIS, and the Hall sensor measurements can be mapped to a distance (such as in millimeters (mm)) of lens shift for the included Hall sensors. An example conversion of Hall sensor measurements to focal length based pixel shift is shown as Equation (2) below:

[0085] OIS shift = UnitConversion * FocalLengthFactor * (a + b * OISLensPos + c * OISLensPos 2 + d * OISLensPos 3 ) (2)

[0087] UnitConversion is the conversion of the Hall sensor reading from a digital representation (such as a particular code for a voltage generated by the Hall sensor) to a distance (such as mm). For example, UnitConversion can be conceptualized in terms of mm units and Hall sensor readings as shown in Equation (2A) below:

[0088]

[0089] FocalLengthFactor is a factor based on the focal length associated with the image sensor. FocalLengthFactor converts distance units (such as mm) from UnitConversion to a length in image sensor pixels of the image sensor. If the distance units of UnitConversion are mm, then FocalLengthFactor can be shown as Equation (2B) below:

[0090]

[0091] While UnitConversion is shown outside of the parentheses to affect all parameters a, b, c, and d, in some implementations, UnitConversion can be included in OISLensPos (thus not affecting parameter a, which can indicate a static offset).

[0092] The conversion can also be affected by the size of the captured frame (the size of the processed frame after EIS) (such as the size of the crop for EIS). In this way, scaling of the conversion can be needed. An example scaling conversion of Hall sensor measurements to focal length based pixel shift is shown as Equation (3) below:

[0093] OIS shift= Scale * UnitConversion * FocalLengthFactor * (a + b * OISLensPos + c * OISLensPos 2 + d * OISLensPos 3 ) (3)

[0095] Scale is a factor to scale a captured frame to a processed frame. In some implementations, the parameters Scale, a, b, c, and d can be determined during device configuration (or otherwise determined prior to operation), stored and used during image stabilization. In some implementations, one or more of the parameters can be determined by the EIS filter module 736 Figure 7 ) that adjusts the parameters of a previously captured frame 740 and evaluates the results of a processed frame that would result from EIS.

[0096] OIS shift may be a value in pixels of a lens position shift (also referred to as a lens shift). In some implementations, OIS shift may indicate a shift from a reference point of a particular pixel coordinate (x, y) of a distorted frame (which is associated with an image sensor pixel (x, y) that captures a frame pixel). A pixel coordinate of a distorted frame can be referred to as C input . The shift associated with a distorted frame pixel coordinate C input may be referred to as OIS shift (C input ).

[0097] A lens position shift can affect lens distortion and rolling shutter distortion. The device 600 can determine a function to reduce lens distortion and rolling shutter distortion. An example reduction function is shown in the following equation (4):

[0098] C corrected = ROT(CorrectionAngle) * LDC(C input + OIS shift (C input )) (4)

[0099] LDC is a lens distortion correction (LDC) transform to convert a distorted frame to a frame that is at least partially undistorted. For example, a lens can compress information intended for edges of a frame toward a center of the frame based on a curvature of the lens. The LDC can be used to stretch a distorted frame back to an intended undistorted frame based on the lens curvature. In some implementations, the LDC can reduce lens distortion, but not completely eliminate lens distortion for every frame. "C input + OIS shift (Cinput ) indicates a shifted pixel coordinate associated with an input pixel coordinate of a frame / image sensor. In this way, the LDC transform is applied to pixel values at shifted pixel coordinates of the frame.

[0100] CorrectionAngle is the difference between angles of the camera 601 during different readout times for a rolling shutter. For example, the CorrectionAngle associated with C input is as shown in equation (5):

[0101] CorrectionAngle(C input ) = Angle(time(C input )) - Angle(time(C center )) (5)

[0102] time(C input ) is the time at which an image sensor pixel C input is read out during a rolling shutter. Angle(time(C input )) is the angle of the camera 601 at time(C input ). C center is the pixel coordinate of the center of the image sensor. time(C center ) is the time at which an image sensor pixel C input is read out during a rolling shutter. Angle(time(C center )) is the angle of the camera 601 at time(C center ). Each angle can include or be represented by one or more Euler angles (which can indicate roll, pitch, or yaw). In this way, measurements of the gyroscope 620 at time(C input ) or time(C center ) can be converted to one or more Euler angles. In another example, each angle can be represented by a quaternion. In this way, measurements of the gyroscope 620 can be converted to quaternions. With reference to Figure 6 , such conversions can be performed by the sensor controller 622, the processor 604, or another suitable component of the device 600. With reference to Figure 7 , such conversions can be performed by the sensor controller 722 or the NCS module 718. In some implementations, the device 600 is calibrated to synchronize or correlate gyroscope measurements and pixel readouts for a rolling shutter. Calibration can include determining which gyroscope measurement corresponds to which portion of the readout for a rolling shutter.

[0103] Referring again to equation (4), ROT(CorrectionAngle) is a rotation transform based on the correction angle for the image sensor pixel. For example, the rotation transform can be a matrix that, when multiplied by a particular image sensor pixel coordinate, indicates the pixel coordinate of a frame that is not distorted by rolling shutter distortion. The device 600 determines the rotation matrix based on all of the pairs of correction angles and pixel coordinates, such that for each image sensor pixel, the rotation matrix is applied to each pixel location of the distorted frame to provide a location (such as a pixel coordinate) in the frame that is not distorted by rolling shutter. If the camera 601 includes a global shutter, the CorrectionAngle for each pixel is zero because all of the pixels in the pixel are captured at one instance in time (and there is no rolling shutter distortion in the capture). In some implementations, when the camera 601 includes a global shutter, ROT(CorrectionAngle) in equation (4) can be replaced with one. In this way, C corrected is the pixel location in the undistorted frame after the LDC transform.

[0104] If the camera 601 includes a rolling shutter, the pixel location provided by the LDC transform and the pixel location provided by the ROT rotation transform are combined to determine the final pixel location C corrected . Equation (4) shows multiplication of the two values to represent the combination, but any suitable operation can be performed to combine the two values. For example, the LDC transform value and the ROT rotation transform value for an input pixel can be averaged. The average can be a simple average, or can be skewed (such as towards one value or the other, or based on edges of the frame of values).

[0105] In some implementations, the device 600 configures the image signal processor 612 to perform EIS by determining pixel values for a processed frame based on the captured frame and a function that reduces lens distortion and rolling shutter distortion. For example, the image signal processor 612 can use the function shown in equation (4) to map pixel values in the captured frame to pixel locations in the processed frame. Referring to Figure 7 , the EIS filter module 736 can determine the function in equation (4) as the EIS transform 738, and the function is used to configure the image signal processor 714.

[0106] While some examples are provided for determining EIS to perform based on lens shift OIS performed during capture, variations of the examples can be used. Thus, the present disclosure is not limited to the specific examples provided.

[0107] Module tilt OIS

[0108] For module tilt OIS, the camera lens position is fixed relative to the image sensor position, and the module including the lens and image sensor can rotate based on rotation of the camera. Referring back to Figure 7 , the lens 704 is fixed in position relative to the position of the image sensor 702. As the module including the lens 704 and image sensor 702 tilts, the lens position sensor 706 can indicate rotation of the lens 704. Because the image sensor 702 tilts (as compared to a lens shift OIS system), determining EIS based on module tilt OIS can include a different function for determining EIS based on lens shift OIS as shown above.

[0109] Referring back to equation (4), module tilt OIS does not have a lens position offset (OIS shift equals zero). In the case of module tilt OIS, OIS shift equals zero, the example reduction function in equation (4) for reducing lens distortion and rolling shutter distortion (by mapping pixels from a distorted frame to pixels of a processed frame) can simplify as shown in the following equation (6):

[0110] C corrected = ROT(CorrectionAngle) * LDC(C input ) (6)

[0111] As described above, the multiplication symbol can refer to any suitable combination of a position value from a rotation transform and a position value from an LDC transform. In some implementations, because the image sensor rotates during module tilt OIS, determining CorrectionAngle can be different as compared to the case based on lens shift OIS. For example, equation (5) showing an example function for CorrectionAngle can not account for rotation of the image sensor during readout for a rolling shutter. For module tilt OIS, the CorrectionAngle function can be adjusted to account for OIS-based rotation of the image sensor. For example, the CorrectionAngle associated with C input for module tilt OIS can be as shown in equation (7):

[0112] CorrectionAngle(C input ) = Angle(time(C ibput )) + Angle OIS (time(C input )) - Angle(time(C center )) - Angle OIS (time(C center )) (7)

[0114] Angle OIS is the angle of the image sensor relative to the camera 601 at time (C input ) or time (C center ). In equation (7), the difference in Angle OIS for the readout time of the input pixel and the readout time of the center pixel is added to the difference in camera angle for the readout time of the input pixel and the readout time of the center pixel. When the image sensor is in a neutral position (such as the module is not tilted to either side, away from the center of the camera 601), Angle OIS is zero. If the image sensor is not tilted (or remains the same tilt) during readout, the difference between Angle OIS in equation (7) is zero, and equation (7) is the same as equation (5).

[0115] In some implementations, the device 600 configures the image signal processor 612 to perform EIS by determining pixel values of a processed frame based on a captured frame and a function that reduces lens distortion and rolling shutter distortion. For example, the image signal processor 612 can use a function shown in equation (6) (which can be based on equation (7)) to map pixel values in a captured frame to pixel positions in a processed frame. With reference to Figure 7 , the EIS filter module 736 can determine the function in equation (6) based on equation (7) as an EIS transform 738, and the function is used to configure the image signal processor 714.

[0116] As described herein, the device 600 can be calibrated prior to use. In some implementations, the calibration can include correlating timestamps from a captured frame (indicating when the frame was captured) with timestamps from lens position sensor measurements (indicating when the lens position sensor 706 (such as a Hall sensor) measured a position of the lens 704). The calibration can also include determining a model of scene motion in a series of captured frames based on the lens position sensor measurements. For the correlated timestamps, the device 600 can determine an offset between the time of the lens position sensor measurement and the time of the captured frame.

[0117] To determine a model of scene motion, the device 600 can determine an image shift function that maps scene motion as a function of the lens position sensor measurements. In this way, the image shift function maps movement of the scene across frames in a series of frames. Figure 9An illustrative flow diagram depicting example operations 900 for calibrating measurements from OIS to be used for EIS is shown. The example operations 900 can be used to determine an image shift function. While examples are described in terms of using Hall sensors, any suitable lens position sensor can be used. Additionally, while the example operations 900 are described in terms of being performed by device 600 or system 700, the example operations 900 can be performed using any suitable configuration of devices, systems, or components.

[0118] At 902, device 600 can capture a series of frames of a calibration scene using camera 601 having a fixed position relative to the calibration scene. In some implementations, the calibration scene is a checkerboard pattern and camera 601 is at a fixed rotation and at a fixed position from the checkerboard pattern. In some implementations, the checkerboard pattern is positioned such that the image sensor captures one or more corners of the checkerboard pattern. Additionally or alternatively, corners of squares in the checkerboard pattern are used for calibration. Other suitable calibration scenes can exist, including shapes with corners (such as rectangles, triangles, etc.), line intersections (such as plus signs, Xs, etc.), or other identifiable aspects in the captured frames (for device 600 to identify points in the scene). As such, the calibration scene can take many forms.

[0119] During capture of the series of frames (902), device 600 can obtain an associated measurement from the lens position sensor for each captured frame (904). For example, device 600 can obtain a measurement from the Hall sensor captured at approximately the same time as the time the scene was captured (such as during readout of one or more pixels of a rolling shutter).

[0120] At 906, device 600 can track one or more portions of the calibration scene in each captured frame. For example, one or more corners of the checkerboard pattern can be tracked, or one or more square corners in the checkerboard pattern can be tracked. To track the one or more portions of the calibration scene (906), device 600 can determine a position of each of the one or more portions of the calibration scene in each captured frame (908).

[0121] At 910, device 600 can determine a capture time for each position based on the indicated frame capture time (such as a timestamp in the frame), a readout duration of the captured frame (based on the rolling shutter), and the position in the captured frame. For a position captured in a frame, and assuming a rolling shutter capture is made per row, an example function for determining a time(position) at which the position was captured is shown in equation (8) below:

[0122]

[0123] Time SOF is the time at which the readout occurs at the beginning of the frame. Duration readout is the duration of time used to read out all of the pixels in the pixel of the image sensor for a rolling shutter. In some implementations, for a rolling shutter, one row can be read out at a time. In this way, duration readout can be multiplied by the vertical position of the location in the frame (x position ) divided by the fraction of the total number of rows in the frame, X.

[0124] At 912, the device 600 determines an image shift function based on the determined capture times. Note that, since the position of the calibration scene is fixed relative to the position of the camera 601, the gyroscope measurements based on camera movement can be ignored. In this way, any movement of the calibration scene in the frames captured during calibration can be associated only with OIS-induced movement (such as movement of the lens 602 or rotation of the lens 602 and image sensor). In this way, any movement of the tracked portion of the calibration scene across the series of frames can be associated with the OIS performed.

[0125] The position can shift in a frame across the series of frames. Thus, the offset from the beginning of the frame of the readout can be different between frames of the position. For example, x position divided by the fraction of the total number of rows, X, can change. In some implementations, to determine the shift function, the device 600 can normalize the times at which the position is read out across the frames. For example, the readout time of the position for each frame can be set to the time associated with the lens displacement sensor measurement time (such as the timestamp for the Hall sensor measurement). The device 600 can associate the most recent timestamp from the Hall sensor measurement with the readout time. In this way, the device 600 can correlate the Hall sensor measurements with each of the times in the time of the position capture of the frame.

[0126] The image shift function can include a curve that fits the lens position sensor measurement data related to the time of the position capture. For example, if four corners of the calibration scene are tracked, then the lens position sensor measurements are correlated with the capture times of one of the corners across the series of frames, resulting in four sets of data. A curve fit is performed to best fit the four sets of data across time. In some implementations, the curve can be defined by a cubic polynomial (although any suitable order can be used).

[0127] As described herein, a lens position sensor can indicate a position of a lens in a two-dimensional space (such as in a plane of image shift OIS or on a curved surface for module tilt OIS). The two-dimensional space can be mapped according to a U-axis and a V-axis. In some implementations of calibration, an image shift function maps lens motion along a vertical axis (X) of an image sensor and a horizontal axis (Y) of the image sensor. The U-axis can be aligned with the X-axis of the image sensor, and the V-axis can be aligned with the Y-axis of the image sensor.

[0128] For a curve fit that spans both the X-axis and the Y-axis, both a first curve for motion on the X-axis and a second curve for motion on the Y-axis are fit during the curve fit (such as via regression). If a third order polynomial curve is used for each axis, a combined polynomial curve for both the X-axis and the Y-axis can be a product of the two third order polynomial curves. An example function of the combined third order polynomial curve is shown in the following equation (9):

[0129]

[0130] SHIFT is a candidate function to map an input pixel position to a shifted pixel position in a frame as a result of OIS during capture. For example, SHIFT can be a candidate combination of OIS shift described herein. An inverse operation of the SHIFT function can be performed on a captured frame to verify a quality of the SHIFT function. The device 600 (or another device) can then determine whether the SHIFT function is an OIS shift based on the quality. For example, the frame after performing the inverse operation of the SHIFT function can be analyzed to determine any remaining warping or shifting (where the quality indicates a reduction in warping or shifting in the frame or an amount of remaining warping or shifting).

[0131] The variable i is an order of a term of the X-axis third order polynomial curve. The variable j is an order of a term of the Y-axis third order polynomial curve. The sum of the variables i and j can reach different maximum values if different orders are used. The parameter p ij is a parameter of a term for each combination of the variables i and j. For example, p 12 is a parameter of the term "Hall(u)*Hall(v) 2 ". Hall(u) is a Hall sensor measurement along the U-axis, and Hall(v) is a Hall sensor measurement along the V-axis. In some implementations, the parameter p ij may be a function of two OIS shiftThe parameters 'a' - 'd' of the function are associated. In comparing equation (1) and equation (9), if the lens position sensor is a Hall sensor, Hall(x) or Hall(y) can be the same as OISLensPos for the X-axis or Y-axis, respectively.

[0132] In addition to performing curve fitting using a combined curve SHIFT of two polynomial curves, curve fitting can involve using two separate polynomial curves in reference to each other for curve fitting. For example, optimization of both curves can be performed simultaneously to determine two separate OIS shift functions.

[0133] In some implementations of curve fitting, one or more parameters can be adjusted, and one or more image shift functions that can be analyzed to determine the quality of the function (such as by analyzing the remaining distortion or shift after performing the inverse of one or more curves on a captured frame). After determining a final image shift function, the device 600 can store the image shift function or parameters of the image shift function to use (such as in memory 606 or another suitable location) to determine EIS to perform based on OIS during frame capture.

[0134] Unless specifically described as being implemented in a particular manner, the techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules or components can also be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques can be realized at least in part by a non-transitory processor-readable storage medium (such as memory 606 in example device 600) comprising instructions 608 that, when executed by a processor 604 (or camera controller 610 or image signal processor 612 or sensor controller 622), cause the device 600 to perform one or more of the methods described above. The non-transitory processor-readable data storage medium can form part of a computer program product, which can include packaging materials. Figure 6

[0135] The non-transitory processor-readable storage medium can include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read only memory (EEPROM), flash memory, other known read-only memory devices, or others. Additionally or alternatively, the techniques can be implemented at least in part by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer or other processor. ​

[0136] The various illustrative logical blocks, modules, circuits, and instructions described in connection with the embodiments disclosed herein can be executed by one or more processors, such as Figure 6 the processor 104 in the example device 600 or the image signal processor 612. Such processors can include, but are not limited to, one or more digital signal processors (DSPs), general purpose microprocessors, application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), field

[0137] While this disclosure has illustrated an illustrative aspect, it will be appreciated that various changes and modifications can be made therein without departing from the scope of the appended claims. Additionally, no aspect of the functionality described herein is required to be performed in any particular order. For example, if performed by the device 600, the camera controller 610, the processor 604, the image signal processor 612, and / or the sensor controller 622, the steps of the described example operations, transformations, and models can be performed in any order and at any frequency. Furthermore, although elements can be described or claimed in the singular, pluralities can be contemplated unless limitation to a single instance is expressly stated. The disclosure is not limited to the illustrative examples set forth and includes any apparatus for performing the functions described herein in aspects of the disclosure.

Claims

1. A device configured to perform electronic image stabilization, comprising: a memory; and one or more processors configured to: obtain optical image stabilization (OIS) information of an optical image stabilization (OIS) performed during capture of a series of frames by an image sensor of a camera, wherein the OIS information comprises indications of positions of a movable camera lens associated with the image sensor during capture of the series of frames; obtain camera position information for the camera; obtain frame capture information for the series of frames; obtain, from the frame capture information, image rendering data for each frame of the series of frames, the image rendering data comprising information for rendering the frames as images; compare, based on the image rendering data, the positions of the movable camera lens to the camera position information; determine that camera movement is greater than a value; and in response to determining that the camera movement is greater than the value, determine, based on the comparison of the positions of the movable camera lens to the camera position information, the series of frames, and the image rendering data, an electronic image stabilization (EIS) filter.

2. The device of claim 1, wherein: the EIS filter comprises a shift function that indicates, for each of a plurality of pixel positions in the series of frames prior to processing, a shift of the pixel position in an unprocessed frame to an associated pixel position in a processed frame.

3. The device of claim 1, wherein: the positions of the movable camera lens are relative to the image sensor; and the one or more processors are further configured to: determine, based on the positions of the movable camera lens during capture of the series of frames, a lens distortion correction (LDC) transform, wherein the EIS filter is based on the LDC transform. the one or more processors are further configured to:

4. The apparatus of claim 1, wherein, determine a rotation transform for reducing rolling shutter distortion, wherein: the rotation transform is based on a difference between an angle of the image sensor at readout of each image sensor pixel and an angle of the image sensor at readout of a center image sensor pixel; and the EIS filter is based on the rotation transform.

5. The device of claim 1, further comprising a camera, the camera comprising: the image sensor configured to capture the series of frames; the movable camera lens associated with OIS during capture of the series of frames; and a lens position sensor associated with the movable camera lens.

6. The device of claim 5, wherein: the OIS is lens shift OIS; and the movable camera lens is configured to move relative to the image sensor.

7. The device of claim 5, wherein: the OIS is module tilt OIS; positions of the movable camera lens relative to positions of the image sensor are fixed; and the movable camera lens is configured to rotate with the image sensor.

8. The device of claim 5, further comprising: ​ ​ a gyroscope configured to measure a position of the camera, wherein the camera position information is based on measurements by the gyroscope.

9. The device of claim 1, further comprising: an image signal processor, wherein the one or more processors are further configured to configure the image signal processor to perform EIS on the series of frames based on the EIS filter.

10. The device of claim 9, further comprising: a video encoder configured to encode the processed series of frames to generate a video after EIS.

11. The apparatus of claim 1, wherein, the one or more processors are further configured to: determine a profile of distortion based on a type of OIS performed; and determine the EIS filter based on an inverse of the determined profile.

12. A method for performing electronic image stabilization (EIS), comprising: obtaining OIS information for optical image stabilization (OIS) performed during capture of a series of frames by an image sensor of a camera, wherein the OIS information includes indications of a position of a movable camera lens associated with the image sensor during capture of the series of frames; obtaining camera position information for the camera; obtaining frame capture information for the series of frames; obtaining image rendering data for each frame of the series of frames from the frame capture information, the image rendering data including information for rendering the frames as images; comparing the position of the movable camera lens to the camera position information based on the image rendering data; determining that camera movement is greater than a value; and in response to determining that the camera movement is greater than the value, determining an electronic image stabilization (EIS) filter based on the comparison of the position of the movable camera lens to the camera position information, the series of frames, and the image rendering data.

13. The method of claim 12, wherein: the EIS filter includes a shift function that indicates, for each of a plurality of pixel positions in the series of frames prior to processing, a shift of the pixel position in an unprocessed frame to an associated pixel position in a processed frame.

14. The method of claim 12, further comprising: determining a lens distortion correction (LDC) transform based on the position of the movable camera lens relative to the image sensor during capture of the series of frames, wherein the EIS filter is based on the LDC transform.

15. The method of claim 12, further comprising determining a rotation transform to reduce rolling shutter distortion, wherein: the rotation transform is based on a difference between an angle of the image sensor at a time of readout of each image sensor pixel and an angle of the image sensor at a time of readout of a center image sensor pixel; and the EIS filter is based on the rotation transform.

16. The method of claim 12, further comprising: capturing the series of frames by the image sensor; and moving the movable camera lens for OIS during capture of the series of frames.

17. The method of claim 16, wherein: ​ The OIS is a lens shift OIS; and Moving the movable camera lens includes moving the movable camera lens relative to the image sensor.

18. The method of claim 16, wherein: The OIS is a module tilt OIS; A position of the movable camera lens is fixed relative to a position of the image sensor; and Moving the movable camera lens includes rotating the movable camera lens with the image sensor.

19. The method of claim 12, further comprising: configuring an image signal processor to perform EIS on the series of frames based on the EIS filter.

20. The method of claim 12, further comprising: determining a profile of distortion based on a type of OIS performed; and determining the EIS filter based on the determined profile in reverse.

21. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a device to perform electronic image stabilization, cause the device to: obtaining OIS information of an optical image stabilization, OIS, performed during the capturing of a series of frames by an image sensor of a camera, wherein, The OIS information includes an indication of a position of a movable camera lens associated with the image sensor during capture of the series of frames; obtain camera position information for the camera; obtain frame capture information for the series of frames; obtain image rendering data for each frame of the series of frames from the frame capture information, the image rendering data including information for rendering the frame as an image; compare the position of the movable camera lens to the camera position information based on the image rendering data; determine that camera movement is greater than a value; and in response to determining that the camera movement is greater than the value, determine an electronic image stabilization (EIS) filter based on the comparison of the position of the movable camera lens to the camera position information, the series of frames, and the image rendering data.

22. The computer readable medium of claim 21, wherein, The EIS filter includes a shift function that indicates, for each of a plurality of pixel positions in the series of frames prior to processing, a shift of the pixel position in an unprocessed frame to an associated pixel position in a processed frame.

23. The computer readable medium of claim 21, wherein, Execution of the instructions further causes the device to determine, during capture of the series of frames, a lens distortion correction (LDC) transform based on a position of the movable camera lens relative to the image sensor, wherein the EIS filter is based on the LDC transform.

24. The computer readable medium of claim 21, wherein, Execution of the instructions further causes the device to determine a rotation transform for reducing rolling shutter distortion, wherein: The rotation transform is based on a difference between an angle of the image sensor at a time of readout of each image sensor pixel and an angle of the image sensor at a time of readout of a center image sensor pixel; and The EIS filter is based on the rotation transform.

25. The computer readable medium of claim 21, wherein, Execution of the instructions further causes the device to: capture the series of frames by the image sensor; and move the movable camera lens relative to the image sensor for OIS during capture of the series of frames, wherein the OIS is a lens shift OIS.

26. The computer readable medium of claim 21, wherein, Execution of the instructions further causes the device to: capture the series of frames by the image sensor; and rotate the movable camera lens for OIS in conjunction with the image sensor during capture of the series of frames, wherein: the OIS is a modular tilt OIS; and a position of the movable camera lens is fixed relative to a position of the image sensor.

27. The computer readable medium of claim 21, wherein, Execution of the instructions further causes the device to: determine a profile of distortion based on a type of OIS performed; and determine the EIS filter based on an inverse of the determined profile.

Images