Dynamic image sensor configuration for improved image stabilization in image capture devices

Abstract

The present disclosure provides systems, methods, and devices for image processing that support improved image quality. In a first aspect, a method of image processing includes determining a scene condition for an image sensor of an image capture device; receiving motion data regarding movement of the image capture device; configuring the image sensor of the image capture device with a first image sensor configuration determined based on the motion data and the scene condition; receiving image data captured with the first image sensor configuration from the image sensor; and determining a video sequence by processing the image data based on the motion data and the scene condition. Other aspects and features are also claimed and described.

Description

Technical Field

[0001] Aspects of the present disclosure relate generally to image processing, and more particularly to improving image quality by reducing unwanted motion artifacts. Some features may enable and provide improved image processing, including improved appearance of photographs of subjects and objects in a scene when the camera is in motion. Background Art

[0002] An image capture device is a device that can capture one or more digital images (whether still images of photographs or image sequences of videos). Capture devices can be incorporated into various devices. By way of example, image capture devices may include stand-alone digital cameras or digital video cameras, wireless communication devices equipped with cameras, such as mobile phones, cellular or satellite radio phones, personal digital assistants (PDAs), tablet or tablet devices, gaming devices, computer devices (such as webcams, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0003] An image capture device captures a sequence of images, where each image corresponds to the appearance of a scene at a particular time, so that when the images are viewed in sequence, the sequence appears as a video showing the motion of objects in the scene. The smoothness of the video depends on the capture rate of the individual images. The faster the individual images are captured, the smoother the video appears because the movement of the objects is better captured as small movements. One frame rate used by image capture devices is 30 frames per second (fps), where images are captured thirty times per second or approximately every 33 milliseconds. Summary of the invention

[0004] The following summarizes some aspects of the present disclosure to provide a basic understanding of the technology discussed. This summary is not an exhaustive overview of all expected features of the present disclosure, and is neither intended to identify key or important elements of all aspects of the present disclosure, nor to delineate the scope of any or all aspects of the present disclosure. Its sole purpose is to present some concepts of one or more aspects of the present disclosure in a summarized form as a prelude to a more detailed description presented later.

[0005] The amount of light that an image sensor can capture is partially proportional to the length of time that the image sensor collects light. At an example rate of 30 frames per second, the amount of time the image sensor collects light is limited to 33 milliseconds. Under certain scene conditions, such as low-light scene conditions in a dark room, this time limitation can result in low-quality images. Reducing the frame rate increases the time that light can be used to capture each image frame in a sequence. For example, the frame duration increases to 66 milliseconds at a frame rate of 15fps. Lower frame rates can improve low-light photography, but lower frame rates result in increased video jitter artifacts, which degrade the appearance of the video. Video jitter artifacts appear as sudden movements when objects exhibit offset positions with no intermediate positions.

[0006] Aspects of the present disclosure allow image sensors to operate at lower frame rates to provide longer frame durations while reducing video jitter artifacts. When certain conditions are met, the frame rate can be reduced and frame rate conversion (FRC) applied to images captured at the reduced frame rate to increase the frames to a higher rate. Using frame rate conversion (FRC) under these conditions allows for improved image capture in, for example, low light conditions, where longer frame durations provide higher signal quality from the image sensor.

[0007] The conditions for entering this mode of operation with lower frame rate image capture and frame rate conversion can be based on motion data, so that this mode of operation is used when motion does not cause video jitter artifacts to increase significantly at lower frame rates. For example, motion data from a sensor in an image capture device can be used to determine when the image capture device is moving slowly or not moving, so that it can be determined that objects in the scene are not moving positions rapidly. As another example, motion data from captured image frames can be used to determine the amount of object motion in the scene, and a low frame rate mode of operation used when objects have limited or low motion in the scene.

[0008] The use of lower frame rates and frame rate conversion improves the tradeoff between signal-to-noise ratio (SNR), frame rate, and motion blur. Dynamic control of the frame rate or other aspects of the image sensor configuration, activation of frame rate conversion, and configuration of frame rate conversion allows the image capture device to advantageously use longer frame durations for image exposure without significantly increasing motion blur or corruption between image frames.

[0009] In one aspect of the present disclosure, a method for image processing includes: determining scene conditions for an image sensor of an image capture device; receiving motion data about movement of the image capture device; configuring the image sensor of the image capture device with a first image sensor configuration determined based on the motion data and the scene conditions; receiving image data captured with the first image sensor configuration from the image sensor; and determining a video sequence by processing the image data based on the motion data and the scene conditions.

[0010] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations including: determining a scene condition for an image sensor of an image capture device; receiving motion data about movement of the image capture device; configuring the image sensor of the image capture device with a first image sensor configuration determined based on the motion data and the scene condition; receiving image data captured with the first image sensor configuration from the image sensor; and determining a video sequence by processing the image data based on the motion data and the scene condition.

[0011] In additional aspects of the present disclosure, an apparatus includes: a component for determining a scene condition for an image sensor of an image capture device; a component for receiving motion data regarding movement of the image capture device; a component for configuring the image sensor of the image capture device with a first image sensor configuration determined based on the motion data and the scene condition; a component for receiving image data captured with the first image sensor configuration from the image sensor; and a component for determining a video sequence by processing the image data based on the motion data and the scene condition.

[0012] In an additional aspect of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations. The operations include: determining a scene condition for an image sensor of an image capture device; receiving motion data regarding movement of the image capture device; configuring the image sensor of the image capture device with a first image sensor configuration determined based on the motion data and the scene condition; receiving image data captured with the first image sensor configuration from the image sensor; and determining a video sequence by processing the image data based on the motion data and the scene condition.

[0013] Image capture devices, i.e., devices that can capture one or more digital images (whether still image photographs or video image sequences), can be incorporated into a variety of devices. By way of example, image capture devices may include stand-alone digital cameras or digital video cameras, wireless communication devices equipped with cameras, such as mobile phones, cellular or satellite radio phones, personal digital assistants (PDAs), tablet or tablet devices, gaming devices, computer devices (such as webcams, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0014] In general, the present disclosure describes image processing techniques for digital cameras having image sensors and image signal processors (ISPs). The ISP may be configured to control the capture of image frames from one or more image sensors and process one or more image frames from the one or more image sensors to generate a view of a scene in a corrected image frame. The corrected image frame may be part of a sequence of image frames that forms a video sequence. A video sequence may include other image frames received from an image sensor or other image sensors and / or other corrected image frames based on input from an image sensor or another image sensor. In some embodiments, processing of one or more image frames may be performed within an image sensor. The image processing techniques described in the embodiments disclosed herein may be performed by circuits in an image sensor, in an image signal processor (ISP), in an application processor (AP), or in a combination of two or all of these components.

[0015] In one example, an image signal processor may receive instructions for capturing a sequence of image frames in response to loading of software (such as a camera application) to generate a preview display from an image capture device. The image signal processor may be configured to generate a single output image frame stream based on image frames received from one or more image sensors. The single output image frame stream may include raw image data from an image sensor, merged image data from an image sensor, or corrected image frames processed by one or more algorithms within the image signal processor. For example, image frames obtained from an image sensor (which may have performed some processing on the data before output to the image signal processor) may be processed in the image signal processor by processing the image frames with an image post-processing engine (IPE) and / or other image processing circuits for performing one or more of tone mapping, portrait lighting, contrast enhancement, gamma correction, etc.

[0016] After the image signal processor determines the output image frame representing the scene using image correction (such as described in various embodiments herein), the output image frame can be displayed on the device display as a single still image and / or as part of a video sequence, saved to a storage device as a picture or video sequence, sent over a network, and / or printed to an output medium. For example, the image signal processor can be configured to obtain input frames of image data (e.g., pixel values) from different image sensors, and then generate corresponding output image frames of the image data (e.g., preview display frames, still image capture, frames for video, frames for object tracking, etc.). In other examples, the image signal processor can output image frames of the image data to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization (e.g., auto focus (AF), automatic white balance (AWB), and automatic exposure control (AEC)), generate video files via output image frames, configure frames for display, configure frames for storage, send frames over a network connection, etc. That is, an image signal processor may obtain incoming frames from one or more image sensors, each coupled to one or more camera lenses, and may then generate and output an output image frame stream to various output destinations.

[0017] In some aspects, a corrected image frame may be produced by combining aspects of the image correction of the present disclosure with other computational photography techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR). In the case of HDR photography, a first image frame and a second image frame are captured using different exposure times, different apertures, different lenses, and / or other characteristics that may result in an improved dynamic range of the fused image when the two image frames are combined. In some aspects, the method may be performed for MFNR photography, wherein the first image frame and the second image frame are captured using the same or different exposure times, and the first image frame and the second image frame are fused to generate a corrected first image frame having reduced noise compared to the captured first image frame.

[0018] In some aspects, a device may include an image signal processor or processor (e.g., an application processor) that includes specific functionality for camera control and / or processing, such as enabling or disabling a merging module or otherwise controlling aspects of image correction. The methods and techniques described herein may be performed entirely by an image signal processor or processor, or various operations may be split between the image signal processor and the processor, and in some aspects split across additional processors.

[0019] The device may include one, two or more image sensors, such as including a first image sensor. When there are multiple image sensors, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a remote image sensor. In another example, the first sensor is configured to obtain an image through a first lens having a first optical axis, and the second sensor is configured to obtain an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. The configuration may occur in a lens cluster on a mobile device, such as where multiple image sensors and associated lenses are located in offset positions on the front or back side of the mobile device. Additional image sensors with larger, smaller, or the same field of view may be included. The image correction techniques described herein may be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0020] In an additional aspect of the present disclosure, a device configured for image processing and / or image capture is disclosed. The device includes a component for capturing an image frame. The device also includes one or more components for capturing data representing a scene, such as an image sensor (including a charge coupled device (CCD), a Bayer filter sensor, an infrared (IR) detector, an ultraviolet (UV) detector, a complementary metal oxide semiconductor (CMOS) sensor), a time-of-flight detector. The device may also include one or more components for gathering and / or focusing light into one or more image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses). These components can be controlled to capture a first image frame and / or a second image frame input to the image processing techniques described herein.

[0021] Other aspects, features, and implementations will become apparent to those of ordinary skill in the art upon reviewing the following description of specific exemplary aspects in conjunction with the accompanying drawings. Although features may be discussed below with respect to certain aspects and drawings, various aspects may include one or more of the advantageous features discussed herein. In other words, while one or more aspects may be discussed as having certain advantageous features, one or more of such features may also be used according to various aspects. In a similar manner, although exemplary aspects may be discussed below as aspects of devices, systems, or methods, exemplary aspects may be implemented in various devices, systems, and methods.

[0022] The method may be embedded in a computer readable medium as a computer program code including instructions for causing a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device comprising: a first network adapter configured to send data, such as an image or video as recorded data or as streaming data, over a first network connection of a plurality of network connections; and a processor coupled to the first network adapter and the memory. The processor may cause the corrected image frame described herein to be sent over a wireless communication network, such as a 5G NR communication network.

[0023] The features and technical advantages of the examples according to the present disclosure have been outlined quite extensively above so that the following specific embodiments may be better understood. Additional features and advantages will be described below. The disclosed concepts and specific examples may be easily used as the basis for modifying or designing other structures for achieving the same purpose of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. When considered in conjunction with the accompanying drawings, the characteristics of the concepts disclosed herein (both their organization and methods of operation) and the associated advantages will be better understood according to the description below. Each of the figures in the accompanying drawings is provided for the purpose of illustration and description, and not as a definition of the limitations of the claims.

[0024] Although various aspects and specific implementations are described in this application by the illustration of some examples, it will be understood by those skilled in the art that additional specific implementations and use cases may be generated in many different arrangements and scenarios. The innovation described herein can be implemented across many different platform types, devices, systems, shapes, sizes, and packaging arrangements. For example, aspects and / or uses can be generated via integrated chip specific implementations and other devices based on non-module components (e.g., end-user devices, vehicles, communication equipment, computing equipment, industrial equipment, retail / purchase equipment, medical equipment, devices that enable artificial intelligence (AI), etc.). Although some examples may or may not specifically point to use cases or applications, the applicability of a wide range of described innovations may occur. The scope of specific implementations can be in the range from chip-level or modular components to non-modular, non-chip-level specific implementations, and further to the scope of aggregated, distributed or original equipment manufacturer (OEM) devices or systems in conjunction with one or more aspects of the described innovations. In some practical environments, the equipment in conjunction with the various aspects and features described may also necessarily include additional components and features for implementing and practicing the various aspects claimed and described. For example, the transmission and reception of wireless signals necessarily include multiple components for analog and digital purposes (e.g., hardware components including antennas, radio frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders / summers, etc.). The innovations described herein are intended to be practiced in a variety of devices, chip-level components, systems, distributed arrangements, end-user devices, etc., having different sizes, shapes, and configurations. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] A further understanding of the nature and advantages of the present disclosure may be achieved by reference to the following drawings. In the drawings, similar components or features may have the same reference label. In addition, various components of the same type may be distinguished by adding a dash and a second label to distinguish between similar components after the reference label. If only the first reference label is used in the specification, the description applies to any of the similar components having the same first reference label, regardless of the second reference label.

[0026] Figure 1 A block diagram of an example device for performing image capture from one or more image sensors in accordance with one or more aspects of the present disclosure is shown.

[0027] Figure 2 A flow chart of an example method for capturing image data with an image sensor configuration based on scene conditions and motion data according to one or more aspects of the present disclosure is shown.

[0028] Figure 3is a flow chart illustrating an example method for frame rate control based on lighting conditions and motion according to some embodiments of the present disclosure.

[0029] Figure 4A A block diagram of an example processing configuration for processing image data according to one or more aspects of the present disclosure is shown.

[0030] Figure 4B A block diagram of an example processing configuration for processing image data according to one or more aspects of the present disclosure is shown.

[0031] Figure 4C A block diagram of an example processing configuration for processing image data according to one or more aspects of the present disclosure is shown.

[0032] Figure 5 is a flow chart illustrating an example determination of an image capture device configuration based on scene conditions and motion criteria in accordance with some aspects of the present disclosure.

[0033] Figure 6 is a flow chart illustrating a method for configuring frame rate conversion according to some aspects of the present disclosure.

[0034] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0035] The specific embodiments described below in conjunction with the accompanying drawings are intended as descriptions of various configurations and are not intended to limit the scope of the present disclosure. Instead, the specific embodiments include specific details for providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in every case, and in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0036] The present disclosure provides systems, devices, methods, and computer-readable media that support improved image processing for image capture operations with an improved balance of image characteristics, including signal-to-noise ratio and video jitter artifacts. The image capture configuration can be dynamically adjusted based on criteria evaluated by the image capture device in controlling image capture and image processing operations. The configuration can be dynamically configured based on a determined movement trajectory of the image capture device and / or scene conditions of the captured field of view. For example, low-light scenes can benefit from longer frame durations (e.g., lower frame rates) for image capture to improve image quality, wherein frame rate conversion (FRC) is applied to the captured lower frame rate image data to increase the frame rate to reduce motion artifacts. Lower frame rates and frame rate conversion can be applied based on determining scene conditions for low-light scenes and determining low motion of the image capture device.

[0037] Specific implementations of the subject matter described in this disclosure may be implemented to achieve one or more of the following potential advantages or benefits. In some aspects, the disclosure provides techniques for improving the appearance of images by providing an SNR at a lower frame rate that is better than the SNR of an image captured at a higher frame rate while providing a similarly smooth video output. Additionally, frame rate conversion may be dynamically configured to reduce breakage artifacts when the scene is dynamic in nature to perform motion interpolation.

[0038] An example device for capturing image frames using one or more image sensors, such as a smart phone, may include a configuration of two, three, four, or more cameras on the back side (e.g., the side opposite the user display) or the front side (e.g., the same side as the user display) of the device. Devices with multiple image sensors include one or more image signal processors (ISPs), computer vision processors (CVPs) (e.g., AI engines), or other suitable circuits for processing images captured by the image sensors. The one or more image signal processors may provide processed image frames to a memory and / or processor (such as an application processor, an image front end (IFE), an image processing engine (IPE), or other suitable processing circuits) for further processing, such as for encoding, storage, transmission, or other manipulation.

[0039] As used herein, an image sensor may refer to the image sensor itself and any specific other components coupled to the image sensor for generating image frames for processing by an image signal processor or other logic circuitry or for storage in a memory (whether a short-term buffer or a long-term non-volatile memory). For example, an image sensor may include other components of a camera, including a shutter, a buffer, or other readout circuitry for accessing individual pixels of the image sensor. An image sensor may also refer to an analog front end or other circuitry for converting an analog signal into a digital representation of an image frame, which is provided to a digital circuit coupled to the image sensor.

[0040] In the following description, a large number of specific details (such as examples of specific components, circuits and processes) are set forth in order to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected or connected through one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific terms are set forth in order to provide a thorough understanding of the present disclosure. However, it will be appreciated by those skilled in the art that these specific details may not be required to implement the teachings disclosed herein. In other examples, in order to avoid obscuring the teachings of the present disclosure, known circuits and devices are shown in block diagram form.

[0041] Some portions of the following detailed description are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, procedures, logic blocks, processes, etc. are conceived as a self-consistent sequence of steps or instructions leading to a desired result. These steps are steps requiring physical manipulations of physical quantities. Typically, although not necessarily, these physical quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0042] In the accompanying drawings, a single box may be described as performing one or more functions. The one or more functions performed by the box may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, the following generally describes each exemplary component, block, module, circuit, and step in terms of its functionality. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Moreover, the example device may include components other than those shown, including known components such as processors, memories, etc.

[0043] Aspects of the present disclosure are applicable to any electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or "frames"). In addition, aspects of the present disclosure can be implemented in image sensors or devices coupled to image sensors having the same or different capabilities and characteristics (such as resolution, shutter speed, sensor type, etc.). In addition, aspects of the present disclosure can be implemented in a device for processing image frames, whether or not the device includes or is coupled to an image sensor, such as a processing device that can retrieve stored images for processing, including a processing device present in a cloud computing system.

[0044] Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout this application, discussions utilizing terms such as "access," "receive," "transmit," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "set," "generate," etc., refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's registers, memories, or other such information storage, transmission, or display devices.

[0045] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (such as a smart phone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more components that can implement at least some portions of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus may include a device or a portion of a device for performing the described operations.

[0046] Figure 1 A block diagram of an example device 100 for performing image capture from one or more image sensors is shown. The device 100 may include or otherwise be coupled to an image signal processor 112 for processing image frames from one or more image sensors, such as a first image sensor 101, a second image sensor 102, and a depth sensor 140. In some implementations, the device 100 also includes or is coupled to a processor 104 and a memory 106 storing instructions 108. The device 100 may also include or be coupled to a display 114 and an input / output (I / O) component 116. The I / O component 116 may be used to interact with a user, such as a touch screen interface and / or physical buttons. The I / O component 116 may also include a network interface for communicating with other devices, including a wide area network (WAN) adapter 152, a local area network (LAN) adapter 153, and / or a personal area network (PAN) adapter 154. An example WAN adapter is a 4G LTE or 5G NR wireless network adapter. An example LAN adapter 153 is an IEEE 802.11 WiFi wireless network adapter. An example PAN adapter 154 is a Bluetooth wireless network adapter. Each of adapters 152, 153, and / or 154 may be coupled to an antenna, including multiple antennas configured for primary and diversity reception and / or configured to receive a particular frequency band. Device 100 may also include or be coupled to a power source 118 for device 100, such as a battery or a component that couples device 100 to an energy source. Device 100 may also include or be coupled to Figure 1 Additional features or components not shown in the figure. In one example, a wireless interface that may include multiple transceivers and a baseband processor may be coupled to or included in the WAN adapter 152 for wireless communication devices. In another example, an analog front end (AFE) for converting analog image frame data to digital image frame data may be coupled between image sensors 101 and 102 and image signal processor 112.

[0047] The device may include or be coupled to a sensor hub 150 for interfacing with sensors to receive data about the movement of the device 100, data about the environment surrounding the device 100, and / or other non-camera sensor data. One example non-camera sensor is a gyroscope, i.e., a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, i.e., a device configured to measure acceleration, which may also be used to determine velocity and distance traveled by appropriately integrating the measured acceleration, and one or more of the acceleration, velocity, and / or distance may be included in the generated motion data. In some aspects, a gyroscope in an electronic image stabilization system (EIS) may be coupled to the sensor hub or directly to the image signal processor 112. In another example, the non-camera sensor may be a global positioning system (GPS) receiver.

[0048] The image signal processor 112 may receive image data such as for forming an image frame. In one embodiment, a local bus connection couples the image signal processor 112 to the image sensors 101 and 102 of the first camera and the second camera, respectively. In another embodiment, a wire interface couples the image signal processor 112 to an external image sensor. In yet another embodiment, a wireless interface couples the image signal processor 112 to the image sensors 101, 102.

[0049] The first camera may include a first image sensor 101 and a corresponding first lens 131. The second camera may include a second image sensor 102 and a corresponding second lens 132. Each of the lenses 131 and 132 may be controlled by an associated auto focus (AF) algorithm 133 executed in the ISP 112, which adjusts the lenses 131 and 132 to focus on a specific focal plane at a certain scene depth from the image sensors 101 and 102. The AF algorithm 133 may be assisted by a depth sensor 140.

[0050] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. The lenses 131 and 132 focus light at the image sensors 101 and 102, respectively, through one or more apertures for receiving light, one or more shutters for blocking light when outside the exposure window, one or more color filter arrays (CFA) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurements into digital information, and / or other suitable components for imaging. The first lens 131 and the second lens 132 can have different fields of view to capture different representations of the scene. For example, the first lens 131 can be an ultra-wide (UW) lens, and the second lens 132 can be a wide (W) lens. Multiple image sensors can include a combination of ultra-wide (high field of view (FOV)) sensors, wide sensors, long-range sensors, and ultra-long-range (low FOV) sensors. That is, each image sensor can be configured by hardware configuration and / or software settings to obtain different but overlapping fields of view. In one configuration, the image sensor is configured with different lenses with different magnifications, which results in different fields of view. The sensors may be configured such that the UW sensor has a larger FOV than the W sensor, the W sensor has a larger FOV than the T sensor, and the T sensor has a larger FOV than the UT sensor. For example, a sensor configured for a wide FOV may capture a field of view in the range of 64 to 84 degrees, a sensor configured for an ultra-lateral FOV may capture a field of view in the range of 100 to 140 degrees, a sensor configured for a long-range FOV may capture a field of view in the range of 10 to 30 degrees, and a sensor configured for an ultra-long-range FOV may capture a field of view in the range of 1 to 8 degrees.

[0051] The image signal processor 112 processes the image frames captured by the image sensors 101 and 102. Figure 1 Device 100 is illustrated as including two image sensors 101 and 102 coupled to image signal processor 112, but any number (e.g., one, two, three, four, five, six, etc.) of image sensors may be coupled to image signal processor 112. In some aspects, a depth sensor such as depth sensor 140 may be coupled to image signal processor 112, and the output from the depth sensor may be processed in a manner similar to that of image sensors 101 and 102. Furthermore, any number of additional image sensors or image signal processors may be present for device 100.

[0052] In some embodiments, the image signal processor 112 may execute instructions from a memory, such as instructions 108 from the memory 106, instructions stored in a separate memory coupled to or included in the image signal processor 112, or instructions provided by the processor 104. In addition or in the alternative, the image signal processor 112 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described in the present disclosure. For example, the image signal processor 112 may include one or more image front ends (IFEs) 135, one or more image post-processing engines 136 (IPEs), and / or one or more automatic exposure compensation (AEC) 134 engines. The AF 133, AEC 134, IFE 135, IPE 136 may each include dedicated circuitry embodied as software code executed by the ISP 112 and / or a combination of hardware within the ISP 112 and software code executed on the ISP. The ISP 112 may additionally execute an automatic white balance (AWB) engine for performing white balance operations. The AWB engine may be executed in, for example, the image front end (IFE) 135 or the ISP 112 or other dedicated or general purpose processing circuitry within the image capture device 100, such as on a digital signal processor (DSP).

[0053] In some implementations, the memory 106 may include a non-transitory or non-transitory computer-readable medium storing computer-executable instructions 108 to perform all or part of one or more operations described in the present disclosure. In some implementations, the instructions 108 include a camera application (or other suitable application) to be executed by the device 100 for generating images or videos. The instructions 108 may also include other applications or programs executed by the device 100, such as an operating system and specific applications other than for image or video generation. For example, the execution of the camera application by the processor 104 may cause the device 100 to generate images using the image sensors 101 and 102 and the image signal processor 112. The memory 106 may also be accessed by the image signal processor 112 to store processed frames, or may be accessed by the processor 104 to obtain processed frames. In some embodiments, the device 100 does not include the memory 106. For example, the device 100 may be a circuit including the image signal processor 112, and the memory may be external to the device 100. The device 100 may be coupled to an external memory and configured to access the memory to write output image frames for display or long-term storage. In some embodiments, device 100 is a system on a chip (SoC) that combines image signal processor 112, processor 104, sensor hub 150, memory 106, and input / output components 116 into a single package.

[0054] In some embodiments, at least one of the image signal processor 112 or the processor 104 executes instructions to perform various operations described herein. For example, execution of the instructions may instruct the image signal processor 112 to start or end capturing an image frame or a sequence of image frames, wherein the capture includes the operations described in the embodiments herein. In some embodiments, the processor 104 may include one or more general-purpose processor cores 104A capable of executing scripts or instructions of one or more software programs (such as instructions 108 stored in the memory 106). For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in the memory 106.

[0055] When executing the camera application, the processor 104 can be configured to instruct the image signal processor 112 to perform one or more operations with reference to the image sensor 101 or 102. For example, the camera application can receive a command to start a video preview display, and upon receiving the command, a video including a sequence of image frames is captured and processed from one or more image sensors 101 or 102. Image correction, such as using a cascaded IPE, can be applied to one or more image frames in the sequence. Execution of instructions 108 by the processor 104 outside the camera application can also cause the device 100 to perform any number of functions or operations. In some embodiments, in addition to the ability to execute software to cause the device 100 to perform multiple functions or operations (such as the operations described herein), the processor 104 can also include an IC or other hardware (e.g., an artificial intelligence (AI) engine 124). In some other embodiments, the device 100 does not include the processor 104, such as when all the described functionality is configured in the image signal processor 112.

[0056] In some embodiments, display 114 may include one or more suitable displays or screens that allow user interaction and / or present items to the user (such as previews of image frames captured by image sensors 101 and 102). In some embodiments, display 114 is a touch-sensitive display. I / O component 116 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from a user and provide output to the user through display 114. For example, I / O component 116 may include (but is not limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (such as a power button), a slider, a switch, etc.

[0057] Although shown coupled to each other via processor 104, components such as processor 104, memory 106, image signal processor 112, display 114, and I / O components 116 may be coupled to each other in various other arrangements, such as via one or more local buses, which are not shown for simplicity. Although image signal processor 112 is illustrated as being separate from processor 104, image signal processor 112 may be a core of processor 104, which is an application processor unit (APU), included in a system on chip (SoC), or otherwise included in processor 104. Although reference is made to device 100 in the examples herein to perform aspects of the present disclosure, some device components may not be present. Figure 1 100. In some embodiments, the present invention is not limited to the configuration of a particular device or component, including device 100.

[0058] Figure 2 A flow chart of an example method for capturing image data with an image sensor configuration based on scene conditions and motion data according to one or more aspects of the present disclosure is shown. Figure 2 Capturing in the image can obtain an improved digital representation of the scene, which results in a photo or video with higher image quality (IQ).

[0059] At box 202, method 200 includes: At box 202, determining a scene condition for a first image sensor of an image capture device. The scene condition may specify, for example, light conditions, environment, location, or other information about global characteristics of a scene in a field of view of the first image sensor. The scene condition may be determined by capturing image data from the image sensor and calculating image statistics about the image data to determine the brightness of the scene. As an example, an image post-processing engine (IPE) 136 may calculate the image statistics and determine a lux index of the scene based on the image statistics, wherein the lux index represents the scene condition. One or more thresholds may be applied to the lux index to characterize the scene as one or more scene conditions. A lux index above a threshold may be characterized as a low light scene condition, while a lux index below a threshold may be characterized as a normal light scene condition.

[0060] Scene conditions may be based on other calculations based on image data. Scene conditions may alternatively or additionally be based on data other than image data. For example, scene conditions may be based on time data, and outdoor lighting conditions may be determined from the time data. One or more thresholds may be applied to the time data to characterize the scene as a daytime scene or a nighttime scene. As another example, scene conditions may be based on location data to characterize scene conditions as indoor scene conditions or outdoor scene conditions. In some embodiments, scene conditions for a first image sensor may be determined based on image data captured by a different second image sensor of an image capture device. For example, when a wide image sensor and a telephoto image sensor have overlapping fields of view, and therefore brightness determined from image data collected by the wide image sensor may be used to determine scene conditions for the telephoto image sensor, and vice versa.

[0061] At box 204, motion data about the movement of the image capture device can be received. The motion data can be IMU data received from one or more sensors, which can include gyroscope data, accelerometer data, magnetometer data, and / or OIS data. The motion data can additionally or alternatively include image data, such as motion vectors calculated from image data captured by a first image sensor. The motion data can be used to determine the motion of the image capture device. For example, the device motion can be inferred to have a motion magnitude based on the motion vector magnitude determined from the image data. As another example, the device motion can be inferred to have a motion magnitude based on the motion magnitude of the gyroscope. As another example, the device motion can be inferred to have a motion magnitude based on the travel distance as recorded by a global positioning system (GPS) or other location determination system. As yet another example, the device motion can be a motion category determined based on sensor data, where the motion category can be walking, running, cycling, sitting, or other activity categories.

[0062] At box 206, the image sensor can be configured based on the motion data received at box 204 and the scene conditions determined at box 202. The image sensor can be configured with a frame duration based on criteria related to the motion data and the scene conditions. The frame duration for the image sensor can be configured by setting a frame rate for image capture performed by the image sensor. The determined configuration can be applied to the image sensor by sending a command to the image sensor and / or setting a configuration register of the image sensor. In some embodiments, when the scene conditions meet a first criterion indicating a low light condition and the motion data meets a second criterion indicating low motion, the frame duration for the image sensor can be increased (such as by reducing the frame rate).

[0063] At block 208, image data is received from the first image sensor while being configured with the first image sensor. For example, the image data may be captured at a frame rate lower than normal operation (such as the higher frame rate configured prior to block 206).

[0064] At box 210, the image data is processed to determine a video sequence, wherein the processing is based on motion data and scene conditions. The processing may include operations in an image front end (IFE) 135, an image post-processing engine (IPE) 136, and / or a frame rate converter (FRC). Features within the IFE, IPE, and / or FRC may be activated or deactivated to correspond to the image sensor configuration set at box 206. For example, when the first image sensor is configured to a longer frame duration (lower frame rate) at box 206, a frame rate converter (FRC) may be enabled at box 210. In some embodiments, when the image sensor is configured to 15fps at box 206, FRC is enabled at box 210, and when the image sensor is configured back to 30fps, FRC is disabled. FRC may be configured to increase the frame rate of image data captured at 15fps to a higher frame rate, i.e., 30fps, 60fps, 90fps, 120fps, or another value higher than 15fps.

[0065] Figure 3 It is shown in Figure 2 An example implementation of the method. Figure 3 300 includes: at block 302, determining that the exposure duration for image capture exceeds the frame duration at the current frame rate. For example, an automatic exposure (AE) algorithm may determine that the desired exposure duration for image capture under current scene conditions is longer than the frame duration at the current frame rate. This may occur when the AE algorithm determines an exposure duration of 44 milliseconds when the current frame rate is 30fps, while the current frame rate only supports a maximum exposure duration of 33 milliseconds. At block 304, low image capture device motion may be determined. For example, a threshold may be applied to gyroscope data to determine whether the device motion is below a threshold amount. At block 306, a new frame rate for image capture may be set for the image sensor to support a longer exposure duration by reducing the frame rate from the current frame rate at block 302. At block 308, image data may be captured at the new frame rate and up-converted to a higher frame rate by a frame rate conversion (FRC) process. In some embodiments, up-conversion returns the processed image data to the frame rate of block 302 prior to the new frame rate set at block 306 .

[0066] FIG. 4A to FIG. 4CThe image capture device is shown in Figure 2 or Figure 3 Some configurations of the method, but the method is not limited to FIG. 4A to FIG. 4C The image capture device is executed.

[0067] Figure 4A A block diagram of an example processing configuration for processing image data according to one or more aspects of the present disclosure is shown. The processor 104 of the system 400 can communicate with the image signal processor (ISP) 112 via a bidirectional bus and / or separate control and data lines. The processor 104 can control the camera 103 via a camera control 410, such as by configuring the camera 103 via a camera driver executed on the processor 104. The configuration may include an image sensor configuration that specifies, for example, frame duration, frame rate, image resolution, readout duration, exposure level, aspect ratio, etc. The camera control 410 may receive motion data from an activity detector 414. The activity detector 414 may receive IMU data from the sensor hub 150, which receives data from one or more sensors. The activity detector 414 may receive other motion data from other components, such as an optical image stabilization (OIS) system of the camera. The activity detector 414 may further receive image data from the camera 103 for determining a motion indicator based on a motion vector from the image data. Activity detector 414 may apply a threshold to the motion data to determine a high motion indicator, which camera control 410 may use to determine the image sensor configuration.

[0068] The camera 103 may obtain image data based on the image sensor configuration. For example, the processor 104 may execute a camera application to: instruct the camera 103 to set a first image sensor configuration for the camera 103; obtain first image data from the camera 103 operating based on the first image sensor configuration; and / or instruct the camera 103 to set a second image sensor configuration when the scene conditions and / or motion data no longer meet the corresponding first and second criteria for operating in a low frame rate capture mode with frame rate conversion. The camera control 410 may also apply an FRC configuration to the FRC 412 in the ISP 112, such as to enable or disable the FRC 412 and / or configure the FRC 412 for a specific conversion technique. The FRC configuration may be applied to the FRC 412 to correspond to the image sensor configuration applied to the camera 103. For example, when the first camera is configured to 15fps, the FRC 412 may be configured to convert the 15fps output of the image data from the camera 103 to 30fps as an output image frame 430 for storage in the memory 106.

[0069] The image data received from the camera 103 may be processed in one or more blocks of the ISP 112 to form an image frame 430, which is stored in the memory 106 and / or provided to the processor 104. The processor 104 may further process the image data to apply effects to the image frame 430. The effects may include Bokeh, lighting, color projection, and / or high dynamic range (HDR) merging. In some embodiments, the functionality for applying effects may be embedded in additional or different components, such as the ISP 112, DSP, ASIC, or other custom logic circuitry for performing additional image processing.

[0070] Figure 2 The method 200 may be configured to perform reference Figure 2 or Figure 3 The operations described above are to determine the output image frame 430. Other configurations for the image capture device may be used to perform Figure 2 or Figure 3 methods such as Figure 4B and Figure 4C In an additional configuration of the present invention, the image capture device performs the following operations: capturing image data at a configured frame rate, and converting the captured image data to a higher frame rate.

[0071] Figure 4B A block diagram of an example processing configuration for processing image data according to one or more aspects of the present disclosure is shown. Figure 4B A system 450 is illustrated, which is similar to Figure 4A The system 400 is similar to the system 400 of FIG. 1 , but the system uses the ISP 112 to determine the configuration and apply the configuration to the image sensor. Figure 4B , the sensor data is provided to an activity detector 414 executed in the ISP 112, which provides information to the camera control 410. The camera control 410 sets the image sensor configuration for the camera 103 and the FRC configuration for the FRC 412. The camera control 410 may also be other aspects of the image processing 316 within the ISP 112, such as by controlling the IFE 135 or IPE 136 in the ISP 112.

[0072] Figure 4C A block diagram of an example processing configuration for processing image data according to one or more aspects of the present disclosure is shown. Figure 4C A system 460 is illustrated, which is similar to Figure 4AThe system 400 is similar to the system 400 of FIG. 1 , but the system includes circuitry external to the ISP 112 and the processor 104 to perform the frame rate conversion operation. For example, the FRC 412 may be a separate logic circuit, including fixed function circuits, programmable logic circuits, or a combination thereof, to perform frame rate conversion on the output of the ISP 112 and store the up-converted image data as output image frames 430 in the memory 106 at a higher frame rate. In other embodiments not shown, the system may be configured similar to Figure 4B The system 450 in which the ISP 112 performs the camera control 410 but the FRC 412 is a separate circuit. Figure 4C Although shown as a separate block and separate circuit in FIG. 4 , FRC 412 may be included in a system on a chip (SoC) or otherwise packaged together with a semiconductor die that includes ISP 112 , processor 104 , and / or memory 106 .

[0073] Figure 5 An example method for determining an image sensor configuration and an FRC configuration of an image capture device is shown in Figure 1 or Figure 2 A portion of the method shown in FIG. 4A to FIG. 4C The system shown in FIG. Figure 5 5 is a flowchart illustrating an example determination of an image capture device configuration based on scene conditions and motion criteria according to some aspects of the present disclosure. Method 500 includes: at box 502, determining whether the scene condition criteria are met. If not, setting a higher frame rate without frame rate conversion at box 508. Scene conditions in which the exposure duration is suitable for the frame duration of the higher frame rate are captured at the higher frame rate without frame rate conversion to provide further reduced video jitter artifacts, thereby improving the image quality of the video sequence. If the scene condition criteria are met at box 502, at box 504, the image capture device determines whether the motion criteria are met. In some embodiments, the motion criteria may include: whether the global motion, such as determined from a gyroscope or other device sensor, is below a global threshold, and whether the local motion, such as determined from a motion vector in a portion of the scene, is below a local threshold. If not, setting a higher frame rate without frame rate conversion at box 508. High motion scenes can be better captured at higher frame rates to improve the smoothness of the video sequence.

[0074] If the scene conditions are met at box 502 and the motion criteria are met at box 504, a lower frame rate and frame rate conversion are set at box 506. Under these criteria, low motion and long exposure, higher signal quality obtained at lower frame rates can produce better image quality, and frame rate conversion reduces some video jitter effects exhibited at lower frame rates. The dynamic operation of the image capture device to control frame rate and frame rate conversion based on motion data and scene conditions improves the overall quality of the captured video sequence by prioritizing certain characteristics at the expense of other characteristics. These trade-offs are configured based on the criteria used at boxes 502 and 504, which may be configurable by the manufacturer of the image capture device through firmware settings and / or by the user of the image capture device through camera application settings.

[0075] When frame rate conversion is activated, such as in Figure 5 At block 508, a method for frame rate conversion may be dynamically controlled based on the scene. Figure 6 6 is a flow chart illustrating a method for configuring frame rate conversion according to some aspects of the present disclosure. Method 600 includes: at box 604, determining whether a scene condition is met. Box 604 may include, for example, determining whether the scene captured by the image sensor is static or dynamic. If the scene is static, the frame rate conversion can be configured to repeat frame mode at box 606. In repeat frame mode, the FRC may duplicate frames to increase the frame rate. Determining whether the scene is static or dynamic can be based on whether it is a local motion value or a global motion value. If the scene is dynamic, the frame rate conversion can be configured to motion interpolation mode at box 608. In motion interpolation mode, the FRC can interpolate a frame between two frames to insert between the two frames.

[0076] In one or more aspects, techniques for supporting image processing may include additional aspects, such as any single aspect or any combination of aspects described below or in conjunction with one or more other processes or devices described elsewhere herein. In a first aspect, supporting image processing may include an apparatus configured to perform operations including: determining a scene condition for an image sensor of an image capture device; receiving motion data regarding movement of the image capture device; configuring the image sensor of the image capture device with a first image sensor configuration determined based on the motion data and the scene condition; receiving image data captured with the first image sensor configuration from the image sensor; and determining a video sequence by processing the image data based on the motion data and the scene condition.

[0077] Additionally, the device may be performed or operated according to one or more aspects as described below. In some specific implementations, the device includes a wireless device, such as a UE or a BS. In some specific implementations, the device may include at least one processor and a memory coupled to the at least one processor. In some specific implementations, the device may include a display and a first camera module, the first camera module includes a first image sensor, wherein the at least one processor is coupled to the first camera and the display. In some specific implementations, the device may include a motion sensor, such as a gyroscope, an accelerometer and / or a compass. The processor may be configured to perform the operations described herein with respect to the device. In some other specific implementations, the device may include a non-transitory computer-readable medium having a program code recorded thereon, and the program code may be executable by a computer to enable the computer to perform the operations described herein with reference to the device. In some specific implementations, the device may include one or more components configured to perform the operations described herein. In some specific implementations, the method of wireless communication may include one or more operations described herein with reference to the device.

[0078] In a second aspect, in combination with the first aspect, configuring the image sensor includes configuring the image sensor with a first readout duration based on the motion data indicating movement of the image capture device satisfying a first criterion.

[0079] In a third aspect, in combination with one or more of the first aspect or the second aspect, the method further comprises: configuring the image sensor with the first readout duration to further indicate a low light scene based on the scene condition.

[0080] In a fourth aspect, in combination with one or more of the first to third aspects, the first criterion includes local motion below a local motion threshold and global motion below a global motion threshold.

[0081] In a fifth aspect, in combination with one or more of the first to fourth aspects, configuring the image sensor with the first readout duration includes configuring the image sensor with a first frame rate.

[0082] In a sixth aspect, in combination with one or more of the first to fifth aspects, processing the image data includes: performing a frame rate conversion (FRC) on the image data to increase the frame rate of the image data, wherein performing the frame rate conversion (FRC) is based on the motion data indicating movement of the image capture device satisfying a first criterion.

[0083] In a seventh aspect, in combination with one or more of the first to sixth aspects, performing frame rate conversion (FRC) includes: when the motion data satisfies a second criterion, repeating frames in the image data to increase a frame rate.

[0084] In an eighth aspect, in combination with one or more of the first to seventh aspects, performing frame rate conversion (FRC) includes: when the motion data does not meet the second criterion, performing motion interpolation on frames in the image data to increase the frame rate.

[0085] In a ninth aspect, in combination with one or more of the first to eighth aspects, configuring the image sensor includes: configuring the image sensor with a first frame rate based on the motion data indicating movement of the image capture device satisfying a first criterion and the scene condition satisfying a second criterion.

[0086] In a tenth aspect, in combination with one or more of the first to ninth aspects, processing the image data includes: based on the motion data indicating the movement of the image capture device satisfying the first criterion and the scene condition satisfying the second criterion, performing frame rate conversion (FRC) on the image data to increase the frame rate of the image data from the first frame rate to a second frame rate higher than the first frame rate.

[0087] In an eleventh aspect, in combination with one or more of the first to tenth aspects, the second criterion is a brightness level below a brightness threshold.

[0088] In a twelfth aspect, in combination with one or more of the first to eleventh aspects, performing frame rate conversion (FRC) includes: when the motion data satisfies a third criterion, repeating frames in the image data to increase a frame rate.

[0089] In a thirteenth aspect, in combination with one or more of the first to twelfth aspects, performing frame rate conversion (FRC) includes: when the motion data does not meet the third criterion, performing motion interpolation on frames in the image data to increase the frame rate.

[0090] It should be understood by those skilled in the art that any of a variety of different technologies and techniques may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0091] This article is about Figures 1 to 6The components, functional blocks and modules include processors, electronic devices, hardware devices, electronic components, logical circuits, memories, software codes, firmware codes, etc., or any combination thereof. Software should be broadly interpreted as meaning instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, and / or functions, etc., whether they are referred to as software, firmware, middleware, microcode, hardware description language, or other terms. In addition, the features discussed herein may be implemented via dedicated processor circuits, via executable instructions, or a combination thereof.

[0092] The technician will further understand that the various exemplary logic boxes, modules, circuits and algorithm steps described in conjunction with the disclosure of this article can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, boxes, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing departure from the scope of the present disclosure. The technician will also easily recognize that the order or combination of components, methods or interactions described herein are only examples, and the components, methods or interactions of various aspects of the present disclosure can be combined or performed in a manner other than the manner illustrated and described herein.

[0093] The various illustrative logics, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0094] The hardware and data processing apparatus for implementing the various illustrative logic elements, logic blocks, modules and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic elements, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller or state machine. In some specific implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. In some specific implementations, specific processes and methods may be performed by circuits specific to a given function.

[0095] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware (including the structures disclosed in this specification and their structural equivalents), or in any combination thereof. Specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device.

[0096] If implemented in software, the function may be stored as one or more instructions or codes on a computer-readable medium or sent via a computer-readable medium. The process of the method or algorithm disclosed herein may be implemented in a processor executable software module that may reside on a computer-readable medium. Computer-readable media include both computer storage media and communication media, and the communication media include any media that can be implemented to transfer a computer program from one place to another. The storage medium may be any available medium that a computer can access. As an example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the required program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection may be appropriately referred to as a computer-readable medium. Disks and optical disks as used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and blue-ray disks, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and instructions on a machine-readable medium and a computer-readable medium, which may be incorporated into a computer program product.

[0097] Various modifications to the specific implementations described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to some other specific implementations without departing from the spirit or scope of the disclosure. Therefore, the claims are not intended to be limited to the specific implementations shown herein, but are to be consistent with the broadest scope consistent with the disclosure, the principles and novel features disclosed herein.

[0098] Additionally, one of ordinary skill in the art will readily recognize that the terms "upper" and "lower" are sometimes used for ease of describing the drawings and indicate relative positions corresponding to the orientation of the drawings on a correctly oriented page, and may not reflect the correct orientation of any device as implemented.

[0099] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be removed from the combination, and a claimed combination may be directed to subcombinations or variations of subcombinations.

[0100] Similarly, although the operations are depicted in a specific order in the figure, this should not be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all the illustrated operations to achieve the desired result. In addition, the accompanying drawings can schematically depict one or more example processes in the form of a flow chart. However, other operations that are not depicted can be combined in the schematically illustrated example process. For example, one or more additional operations can be performed before, after, simultaneously or between any illustrated operations. In some environments, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the specific implementation described above should not be understood as requiring such separation in all specific implementations, and it should be understood that the program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other specific implementations also fall within the scope of the appended claims. In some cases, the actions narrated in the claims can be performed in different orders and still achieve the desired result.

[0101] As used herein, including in the claims, the term "or" used in a list of two or more items means that any of the listed items may be employed alone, or any combination of two or more of the listed items may be employed. For example, if a composition is described as containing components A, B, or C, the composition may contain A alone; B alone; C alone; A and B combined; A and C combined; B and C combined; or A, B, and C combined. In addition, as used herein, including in the claims, "or" as used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination of any of these. The term "substantially" is defined as to a large extent, but not necessarily entirely, what is specified (and includes what is specified; for example, substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by one of ordinary skill in the art. In any disclosed implementation, the term "substantially" may be replaced with "within [percent] of" a specified content, where the percentage includes 0.1%, 1%, 5%, or 10%.

[0102] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Therefore, the disclosure is not intended to be limited to the examples and designs described herein, but to be consistent with the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method, comprising: determining a scene condition for an image sensor of an image capture device; receiving motion data regarding movement of the image capture device; configuring the image sensor of the image capture device with a first image sensor configuration determined based on the motion data and the scene condition; receiving, from the image sensor, image data captured with the first image sensor configuration; as well as A video sequence is determined by processing the image data based on the motion data and the scene condition.

2. The method according to claim 1, wherein: Configuring the image sensor includes configuring the image sensor with a first readout duration based on the motion data indicative of movement of the image capture device satisfying a first criterion. 3 . The method of claim 2 , wherein configuring the image sensor with the first readout duration is further based on the scene condition indicating a low-light scene. The method of claim 2 , wherein the first criterion comprises local motion below a local motion threshold and global motion below a global motion threshold. 5 . The method of claim 2 , wherein configuring the image sensor with the first readout duration comprises configuring the image sensor with a first frame rate.

6. The method of claim 1 , wherein processing the image data comprises: Frame rate conversion (FRC) is performed on the image data to increase a frame rate of the image data, wherein performing the frame rate conversion (FRC) is based on the motion data indicating movement of the image capture device satisfying a first criterion.

7. The method according to claim 6, wherein: Performing frame rate conversion (FRC) includes: repeating frames in the image data to increase a frame rate when the motion data satisfies a second criterion; and Performing frame rate conversion (FRC) includes: when the motion data does not meet the second criterion, performing motion interpolation on frames in the image data to increase a frame rate.

8. The method according to claim 1, wherein: Configuring the image sensor includes configuring the image sensor with a first frame rate based on the motion data indicating movement of the image capture device satisfying a first criterion and the scene condition satisfying a second criterion, and Processing the image data includes: based on the motion data indicating movement of the image capture device satisfying the first criterion and the scene condition satisfying the second criterion, performing frame rate conversion (FRC) on the image data to increase the frame rate of the image data from the first frame rate to a second frame rate higher than the first frame rate. The method of claim 8 , wherein the second criterion is a brightness level below a brightness threshold.

10. The method according to claim 9, wherein: Performing frame rate conversion (FRC) includes: repeating frames in the image data to increase a frame rate when the motion data satisfies a third criterion; and Performing frame rate conversion (FRC) includes: when the motion data does not meet the third criterion, performing motion interpolation on frames in the image data to increase a frame rate.

11. A device, comprising: a memory storing processor readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor readable code to cause the at least one processor to perform operations including: determining a scene condition for an image sensor of an image capture device; receiving motion data regarding movement of the image capture device; configuring the image sensor of the image capture device with a first image sensor configuration determined based on the motion data and the scene condition; receiving, from the image sensor, image data captured with the first image sensor configuration; as well as A video sequence is determined by processing the image data based on the motion data and the scene condition.

12. The device according to claim 11, wherein: Configuring the image sensor includes configuring the image sensor with a first readout duration based on the motion data indicative of movement of the image capture device satisfying a first criterion.

13. The apparatus of claim 12, wherein configuring the image sensor with the first readout duration further indicates a low-light scene based on the scene condition.

14. The apparatus of claim 12, wherein the first criterion comprises local motion below a local motion threshold and global motion below a global motion threshold.

15. The apparatus of claim 12, wherein configuring the image sensor with the first readout duration comprises configuring the image sensor with a first frame rate.

16. The apparatus of claim 11, wherein processing the image data comprises: Frame rate conversion (FRC) is performed on the image data to increase a frame rate of the image data, wherein performing the frame rate conversion (FRC) is based on the motion data indicating movement of the image capture device satisfying a first criterion.

17. The device according to claim 16, wherein: Performing frame rate conversion (FRC) includes: repeating frames in the image data to increase a frame rate when the motion data satisfies a second criterion; and Performing frame rate conversion (FRC) includes: when the motion data does not meet the second criterion, performing motion interpolation on frames in the image data to increase a frame rate.

18. The apparatus of claim 11, wherein: Configuring the image sensor includes configuring the image sensor with a first frame rate based on the motion data indicating movement of the image capture device satisfying a first criterion and the scene condition satisfying a second criterion, and Processing the image data includes: based on the motion data indicating movement of the image capture device satisfying the first criterion and the scene condition satisfying the second criterion, performing frame rate conversion (FRC) on the image data to increase the frame rate of the image data from the first frame rate to a second frame rate higher than the first frame rate. The apparatus of claim 18 , wherein the second criterion is a brightness level below a brightness threshold.

20. The apparatus of claim 19, wherein: Performing frame rate conversion (FRC) includes: repeating frames in the image data to increase a frame rate when the motion data satisfies a third criterion; and Performing frame rate conversion (FRC) includes: when the motion data does not meet the third criterion, performing motion interpolation on frames in the image data to increase a frame rate.

21. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising: determining a scene condition for an image sensor of an image capture device; receiving motion data regarding movement of the image capture device; configuring the image sensor of the image capture device with a first image sensor configuration determined based on the motion data and the scene condition; receiving, from the image sensor, image data captured with the first image sensor configuration; as well as A video sequence is determined by processing the image data based on the motion data and the scene condition.

22. The non-transitory computer readable medium of claim 21, wherein: Configuring the image sensor includes configuring the image sensor with a first readout duration based on the motion data indicative of movement of the image capture device satisfying a first criterion.

23. The non-transitory computer readable medium of claim 21, wherein: Configuring the image sensor includes configuring the image sensor with a first frame rate based on the motion data indicating movement of the image capture device satisfying a first criterion and the scene condition satisfying a second criterion, and Processing the image data includes: based on the motion data indicating movement of the image capture device satisfying the first criterion and the scene condition satisfying the second criterion, performing frame rate conversion (FRC) on the image data to increase the frame rate of the image data from the first frame rate to a second frame rate higher than the first frame rate.

24. The non-transitory computer readable medium of claim 23, wherein the second criterion is a brightness level below a brightness threshold.

25. The non-transitory computer readable medium of claim 24, wherein: Performing frame rate conversion (FRC) includes: repeating frames in the image data to increase a frame rate when the motion data satisfies a third criterion; and Performing frame rate conversion (FRC) includes: when the motion data does not meet the third criterion, performing motion interpolation on frames in the image data to increase a frame rate.

26. An image capturing device, the image capturing device comprising: monitor; Motion sensor; a first camera, the first camera comprising a first image sensor; a memory storing processor readable code; and at least one processor coupled to the memory, the motion sensor, the display, and the first camera, the at least one processor configured to execute the processor readable code to cause the at least one processor to perform operations including: determining a scene condition for the first image sensor; receiving motion data based on output from the motion sensor, the motion data indicating movement of the image capture device; determining an image capture configuration, the image capture configuration comprising: a first image sensor configuration, the first image sensor configuration comprising a frame rate value; and a frame rate conversion (FRC) configuration, the image capture configuration being based on the motion data and the scene condition; configuring the first image sensor based on the image capture configuration; receiving, from the first image sensor, image data captured with the first image sensor configuration; and A video sequence is determined by processing the image data based on the image capture configuration.

27. The image capture device of claim 26, wherein determining the image capture configuration comprises: The frame rate value is determined as a first frame rate based on: The motion data indicative of movement of the image capture device satisfies a first criterion, and The brightness level of the scene of the first image sensor meets a second criterion.

28. The image capture device of claim 27, wherein determining the image capture configuration comprises: Determining that frame rate conversion (FRC) is included when processing the image data is based on: The motion data indicative of movement of the image capture device satisfies a first criterion, and The brightness level of the scene of the first image sensor meets a second criterion.

29. The image capture device of claim 28, wherein determining the image capture configuration comprises: When the motion data satisfies a third criterion, determining to repeat frames for frame rate conversion (FRC) in the image data to increase a frame rate; as well as When the motion data does not satisfy the third criterion, it is determined to perform motion interpolation on frames for frame rate conversion (FRC) in the image data to increase a frame rate.

30. The image capture device of claim 28, further comprising: The video sequence is displayed on the display, wherein the video sequence has a second frame rate that is higher than the first frame rate.

Images