Variable aperture (VA) camera control for controlling depth of focus

Abstract

This disclosure provides systems, methods, and devices for image capture and image processing that support capture using a variable aperture (VA) camera. In a first aspect, an image processing method includes determining a plurality of depths for a scene, the plurality of depths corresponding to a plurality of regions of interest; determining an aperture size of the camera based on the plurality of depths; determining a lens position of the camera based on the aperture size and the plurality of depths; and controlling the camera to adjust a variable aperture (VA) of the camera based on the aperture size and to adjust a lens associated with the variable aperture (VA) based on the lens position. Other aspects and features are also claimed and described.

Description

Technical Field

[0001] Aspects of the present disclosure relate generally to image capture devices, and more particularly to control of camera configurations for obtaining image data. Some features may enable and provide improved image capture control, including the use of a variable aperture (VA) to obtain a desired depth of focus from a camera. Background Art

[0002] An image capture device is a device that can capture one or more digital images (whether still images for photographs or image sequences for videos). Capture devices can be incorporated into various devices. By way of example, image capture devices may include stand-alone digital cameras or digital video cameras, wireless communication devices equipped with cameras, such as mobile phones, cellular or satellite radio phones, personal digital assistants (PDAs), tablet or tablet devices, gaming devices, computer devices (such as web cameras, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0003] In some scenes, the photographer may wish to direct the viewer's focus to a portion of the scene. For example, in a portrait photo of a person, the photographer may want the viewer to focus on the person rather than the rest of the scenery. The photographer may choose a low aperture lens for such a photo because the low aperture causes objects at a different depth than the person to be noticeably blurred. Changing the aperture size typically requires replacing components of the camera, but changing components of a camera is a slow process. Summary of the invention

[0004] The following summarizes some aspects of the present disclosure to provide a basic understanding of the technology discussed. This summary is not an exhaustive overview of all expected features of the present disclosure, and is neither intended to identify key or important elements of all aspects of the present disclosure, nor to define the scope of any or all aspects of the present disclosure. Its only purpose is to give some concepts of one or more aspects of the present disclosure in a summarized form as a prelude to more specific embodiments presented later.

[0005] In some aspects, a variable aperture (VA) may be included in a camera of an image capture device. The variable aperture (VA) may be controlled by an electronic signal without having to replace the camera of the image capture device. The variable aperture (VA) may be controlled during operation of the camera to obtain different aperture sizes corresponding to a predetermined standard. The standard may include a desired depth of focus based on the depth of multiple regions of interest of a scene. The desired depth of focus may be determined to obtain an aperture size with a depth of focus, thereby obtaining an image frame with multiple regions of interest in a focused state. The multiple regions of interest may include multiple people, so that the variable aperture (VA) is controlled to obtain the desired depth of focus of the scene. The lens position of the camera may also be determined to configure the depth of focus of the camera. Including a variable aperture (VA) in the camera and controlling the aperture size and lens position of the camera may improve the quality of the image obtained from the camera.

[0006] In one aspect of the present disclosure, an image processing method includes: determining multiple depths for a scene, the multiple depths corresponding to multiple regions of interest; determining an aperture size of a camera based on the multiple depths; determining a lens position of the camera based on the aperture size and the multiple depths; and controlling the camera to adjust a variable aperture (VA) of the camera based on the aperture size and adjust a lens associated with the variable aperture (VA) based on the lens position.

[0007] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to: determine a plurality of depths for a scene, the plurality of depths corresponding to a plurality of regions of interest; determine an aperture size of a camera based on the plurality of depths; determine a lens position of the camera based on the aperture size and the plurality of depths; and control the camera to adjust a variable aperture (VA) of the camera based on the aperture size and adjust a lens associated with the variable aperture (VA) based on the lens position.

[0008] In additional aspects of the present disclosure, an apparatus includes: means for determining multiple depths for a scene, the multiple depths corresponding to multiple regions of interest; means for determining an aperture size of a camera based on the multiple depths; means for determining a lens position of the camera based on the aperture size and the multiple depths; and means for controlling the camera to adjust a variable aperture (VA) of the camera based on the aperture size and to adjust a lens associated with the variable aperture (VA) based on the lens position.

[0009] In an additional aspect of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations including: determining a plurality of depths for a scene, the plurality of depths corresponding to a plurality of regions of interest; determining an aperture size of a camera based on the plurality of depths; determining a lens position of the camera based on the aperture size and the plurality of depths; and controlling the camera to adjust a variable aperture (VA) of the camera based on the aperture size and adjust a lens associated with the variable aperture (VA) based on the lens position.

[0010] The image processing methods described herein may be performed by an image capture device and / or on image data captured by one or more image capture devices. Image capture devices (devices that can capture one or more digital images, whether still images, photographs or sequences of images of video) may be incorporated into various devices. By way of example, image capture devices may include stand-alone digital cameras or digital video cameras, wireless communication devices equipped with cameras, such as mobile phones, cellular or satellite radio phones, personal digital assistants (PDAs), panels or tablet devices, gaming devices, computer devices (such as web cameras, video surveillance cameras) or other devices with digital imaging or video capabilities.

[0011] The image processing techniques described herein may involve a digital camera having an image sensor and processing circuits (e.g., an application specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a central processing unit (CPU)). An image signal processor (ISP) may include one or more of these processing circuits and be configured to perform operations to obtain image data for processing according to the image processing techniques described herein and / or involved in the image processing techniques described herein. The ISP may be configured to control the capture of image frames from one or more image sensors and determine one or more image frames from the one or more image sensors to generate a view of a scene in an output image frame. The output image frame may be part of a sequence of image frames that forms a video sequence. The video sequence may include other image frames received from the image sensor or other image sensors.

[0012] In an example application, an image signal processor (ISP) may receive instructions for capturing a sequence of image frames in response to loading of software (such as a camera application) to generate a preview display from an image capture device. The image signal processor may be configured to generate a single output image frame stream based on image frames received from one or more image sensors. The single output image frame stream may include raw image data from an image sensor, merged image data from an image sensor, or corrected image data processed by one or more algorithms within the image signal processor. For example, the image frames may be processed by an image post-processing engine (IPE) and / or other image processing circuits to process image frames obtained from an image sensor in an image signal processor (which may have performed some processing on the data before being output to the image signal processor) to perform one or more of tone mapping, portrait lighting, contrast enhancement, gamma correction, etc. The output image frames from the ISP may be stored in a memory and retrieved by an application processor executing a camera application, which may perform further processing on the output image frames to adjust the appearance of the output image frames and reproduce the output image frames on a display for viewing by a user.

[0013] After the output frame representing the scene is determined by the image signal processor and / or the application processor (such as by the image processing techniques described in various embodiments herein), the output image frame may be displayed on the device display as a single still image and / or as part of a video sequence, saved to a storage device as a picture or video sequence, sent over a network, and / or printed to an output medium. For example, an image signal processor (ISP) may be configured to obtain an input frame of image data (e.g., pixel values) from one or more image sensors, and then generate corresponding output image frames (e.g., preview display frames, still image capture, frames for video, frames for object tracking, etc.). In other examples, the image signal processor may output the image frames to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization (e.g., auto focus (AF), automatic white balance (AWB), and automatic exposure control (AEC)), generate video files via output frames, configure frames for display, configure frames for storage, send frames over a network connection, etc. In general, an image signal processor (ISP) may obtain incoming frames from one or more image sensors, generate an output frame stream, and output the output frame stream to various output destinations.

[0014] In some aspects, the output image frame may be generated by combining aspects of the image correction of the present disclosure with other computational photography techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR). In the case of HDR photography, the first image frame and the second image frame are captured using different exposure times, different apertures, different lenses, and / or other characteristics that may result in improved dynamic range of the fused image when the two image frames are combined. In some aspects, the method may be performed for MFNR photography, wherein the first image frame and the second image frame are captured using the same or different exposure times, and the first image frame and the second image frame are fused to generate a corrected first image frame having reduced noise compared to the captured first image frame.

[0015] In some aspects, a device may include an image signal processor or processor (e.g., an application processor) that includes specific functionality for camera control and / or processing, such as enabling or disabling a merging module or otherwise controlling aspects of image correction. The methods and techniques described herein may be performed entirely by an image signal processor or processor, or various operations may be split between the image signal processor and the processor, and in some aspects split across additional processors.

[0016] The device may include one, two or more image sensors, such as including a first image sensor. When there are multiple image sensors, the configurations of these image sensors may be different. For example, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a remote image sensor. In another example, the first sensor is configured to obtain an image through a first lens having a first optical axis, and the second sensor is configured to obtain an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. Any of these or other configurations may be part of a lens cluster on a mobile device, such as where multiple image sensors and associated lenses are located at offset positions on the front or back side of the mobile device. Additional image sensors with larger, smaller or the same field of view may be included. The image processing techniques described herein may be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0017] In an additional aspect of the present disclosure, a device configured for image processing and / or image capture is disclosed. The device includes components for capturing image frames. The device also includes one or more components for capturing data representing a scene, such as an image sensor (including a charge coupled device (CCD), a Bayer filter sensor, an infrared (IR) detector, an ultraviolet (UV) detector, a complementary metal oxide semiconductor (CMOS) sensor), a time-of-flight detector. The device may also include one or more components for gathering and / or focusing light into one or more image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses). These components can be controlled to capture a first image frame and / or a second image frame input to the image processing techniques described herein.

[0018] For those of ordinary skill in the art, other aspects, features, and specific implementations will become apparent when reviewing the following description of specific exemplary aspects in conjunction with the accompanying drawings. Although each feature may be discussed below with respect to certain aspects and accompanying drawings, each aspect may include one or more of the advantageous features discussed herein. In other words, although one or more aspects may be discussed as having certain advantageous features, one or more of such features may also be used according to each aspect. In a similar manner, although the exemplary aspects may be discussed below as aspects of a device, system, or method, the exemplary aspects may be implemented in various devices, systems, and methods.

[0019] The method may be embedded in a computer readable medium as a computer program code including instructions for causing a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device comprising: a first network adapter configured to send data, such as an image or video as recorded data or as streaming data, over a first network connection of a plurality of network connections; and a processor coupled to the first network adapter and a memory. The processor may cause the output image frame described herein to be sent over a wireless communication network, such as a 5G NR communication network.

[0020] The features and technical advantages of examples according to the present disclosure have been outlined quite extensively above so that the following specific embodiments may be better understood. Additional features and advantages will be described below. The disclosed concepts and specific examples may be easily used as a basis for modifying or designing other structures for the same purpose of achieving the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. When considered in conjunction with the accompanying drawings, the characteristics of the concepts disclosed herein (both their organization and methods of operation) and the associated advantages will be better understood according to the following description. Each of the figures in the accompanying drawings is provided for the purpose of illustration and description, and not as a definition of limitations to the claims.

[0021] Although various aspects and specific implementations are described in this application by the illustration of some examples, it will be understood by those skilled in the art that additional specific implementations and use cases may be generated in many different arrangements and scenarios. The innovation described herein can be implemented across many different platform types, devices, systems, shapes, sizes, and packaging arrangements. For example, aspects and / or uses can be generated via integrated chip specific implementations and other devices based on non-module components (e.g., end-user devices, vehicles, communication equipment, computing equipment, industrial equipment, retail / purchase equipment, medical equipment, devices that enable artificial intelligence (AI), etc.). Although some examples may or may not specifically point to use cases or applications, the applicability of various types of described innovations may occur. The scope of specific implementations can range from chip-level or module components to non-module, non-chip-level specific implementations, and further to aggregated, distributed, or original equipment manufacturer (OEM) devices or systems in conjunction with one or more aspects of the described innovations. In some practical environments, the equipment in conjunction with the various aspects and features described may also necessarily include additional components and features for implementing and practicing the various aspects claimed and described. For example, the transmission and reception of wireless signals necessarily include multiple components for analog and digital purposes (e.g., hardware components including antennas, radio frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders / summers, etc.). The innovations described herein are intended to be practiced in a variety of devices, chip-level components, systems, distributed arrangements, end-user devices, etc., having different sizes, shapes, and configurations. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] A further understanding of the nature and advantages of the present disclosure may be achieved by reference to the following drawings. In the drawings, similar components or features may have the same reference label. In addition, various components of the same type may be distinguished by adding a dash and a second label to distinguish between similar components after the reference label. Where only the first reference label is used in the specification, the description applies to any of the similar components having the same first reference label, regardless of the second reference label.

[0023] Figure 1 A block diagram of an example device for performing image capture from one or more image sensors is shown.

[0024] Figure 2 is a block diagram illustrating an example data flow path for image data processing in an image capture device according to one or more embodiments of the present disclosure.

[0025] Figure 3is a graphical representation of a variable aperture at several aperture sizes that may be controlled by a camera configuration according to one or more embodiments of the present disclosure.

[0026] Figure 4 A flowchart is shown of an example method for controlling a camera to capture an image with a desired portion of a scene in focus, according to some embodiments of the present disclosure.

[0027] Figure 5 is a block diagram illustrating an example processor configuration for capturing image data using a variable aperture (VA) camera in accordance with one or more embodiments of the present disclosure.

[0028] Figure 6 is a graphical representation of camera control of a variable aperture (VA) for an example scene, in accordance with one or more embodiments of the present disclosure.

[0029] Figure 7 is a graphical representation of lens position control operations for a variable aperture (VA) camera according to one or more embodiments of the present disclosure.

[0030] Figure 8 is a graphical representation of image data captured from a variable aperture (VA) camera in accordance with one or more embodiments of the present disclosure.

[0031] Like reference numbers and designations in different drawings indicate like elements. DETAILED DESCRIPTION

[0032] The specific embodiments described below in conjunction with the accompanying drawings are intended as descriptions of various configurations and are not intended to limit the scope of the present disclosure. Instead, the specific embodiments include specific details for providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in every case, and in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0033] The present disclosure provides systems, devices, methods, and computer-readable media that support image capture operations and subsequent image processing operations, including techniques for controlling the configuration of a camera of an image capture device. The camera may include a variable aperture (VA). The camera configuration may specify the aperture size of the variable aperture (VA) and the lens position of the lens to set a focus depth around which the focus depth of a scene is centered.

[0034] Specific implementations of the subject matter described in the present disclosure may be implemented to achieve one or more of the following potential advantages or benefits. In certain aspects, the present disclosure provides techniques for capturing images of multiple objects (such as multiple people) in focus. The techniques may also provide optimized depth of focus and lens position for a camera. Obtaining images with sharp details of multiple regions of interest from a camera may reduce or eliminate post-processing to deblur the regions of interest, thereby reducing computation time and reducing battery consumption of the image capture device.

[0035] Example devices for capturing image frames using one or more image sensors, such as smart phones, may include a configuration of one, two, three, four, or more cameras on the back side (e.g., the side opposite the primary user display) and / or the front side (e.g., the same side as the primary user display) of the device. These devices may include one or more image signal processors (ISPs), computer vision processors (CVPs) (e.g., AI engines), or other suitable circuits for processing images captured by the image sensors. The one or more image signal processors (ISPs) may store the output image frames in a memory and / or provide the output image frames to processing circuitry in other ways (such as via a bus). The processing circuitry may perform further processing, such as encoding, storing, transmitting, or other manipulations of the output image frames.

[0036] As used herein, an image sensor may refer to the image sensor itself and any specific other components coupled to the image sensor for generating image frames for processing by an image signal processor or other logic circuitry or for storage in a memory (whether a short-term buffer or a long-term non-volatile memory). For example, an image sensor may include other components of a camera, including a shutter, a buffer, or other readout circuitry for accessing individual pixels of the image sensor. An image sensor may also refer to an analog front end or other circuitry for converting an analog signal into a digital representation of an image frame, which is provided to a digital circuit coupled to the image sensor.

[0037] In the description of the embodiments herein, a large amount of specific details (such as examples of specific components, circuits and processes) are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupling" means directly connected or connected by one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific terms are set forth to provide a thorough understanding of the present disclosure. However, it will be appreciated by those skilled in the art that the teachings disclosed herein may not require these specific details. In other examples, in order to avoid blurring the teachings of the present disclosure, known circuits and equipment are shown in block diagram form.

[0038] Some portions of the following detailed description are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, procedures, logic blocks, processes, etc. are conceived as a self-consistent sequence of steps or instructions leading to a desired result. These steps are steps requiring physical manipulations of physical quantities. Typically, although not necessarily, these physical quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0039] In the accompanying drawings, a single box may be described as performing one or more functions. The one or more functions performed by the box may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, the following generally describes various exemplary components, blocks, modules, circuits, and steps in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Moreover, the example device may include components other than those components shown, including known components such as processors, memories, etc.

[0040] Aspects of the present disclosure are applicable to any electronic device that includes, is coupled to, or otherwise processes data from one, two, or more image sensors capable of capturing image frames (or "frames"). The terms "output image frame" and "corrected image frame" may refer to, in addition, aspects of the present disclosure may be implemented in image sensors or devices coupled to image sensors having the same or different capabilities and characteristics (such as resolution, shutter speed, sensor type, etc.). In addition, aspects of the present disclosure may be implemented in a device for processing image frames, whether or not the device includes or is coupled to an image sensor, such as a processing device that can retrieve stored images for processing, including a processing device present in a cloud computing system.

[0041] Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout this application, discussions utilizing terms such as "access," "receive," "transmit," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "set," "generate," etc., refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's registers, memories, or other such information storage, transmission, or display devices.

[0042] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (e.g., a smartphone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some portions of the present disclosure. Although the description and examples herein use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus may include a device or a portion of a device for performing the described operations.

[0043] Certain components of a device or apparatus described as “means for accessing,” “means for receiving,” “means for sending,” “means for using,” “means for selecting,” “means for determining,” “means for normalizing,” “means for multiplying,” or other similarly named terms referring to one or more operations on data (such as image data) may refer to processing circuits (e.g., application specific integrated circuits (ASICs), digital signal processors (DSPs), graphics processing units (GPUs), central processing units (CPUs)) configured to perform the functions through hardware, software, or a combination of hardware configured by software.

[0044] Figure 1A block diagram of an example device 100 for performing image capture from one or more image sensors is shown. The device 100 may include or otherwise be coupled to an image signal processor 112 for processing image frames from one or more image sensors, such as a first image sensor 101, a second image sensor 102, and a depth sensor 140. In some implementations, the device 100 also includes or is coupled to a processor 104 and a memory 106 storing instructions 108. The device 100 may also include or be coupled to a display 114 and an input / output (I / O) component 116. The I / O component 116 may be used to interact with a user, such as a touch screen interface and / or physical buttons. The I / O component 116 may also include a network interface for communicating with other devices including a wide area network (WAN) adapter 152, a local area network (LAN) adapter 153, and / or a personal area network (PAN) adapter 154. An example WAN adapter is a 4G LTE or 5G NR wireless network adapter. An example LAN adapter 153 is an IEEE 802.11 WiFi wireless network adapter. An example PAN adapter 154 is a Bluetooth wireless network adapter. Each of adapters 152, 153, and / or 154 may be coupled to an antenna, including multiple antennas configured for primary and diversity reception and / or configured to receive a particular frequency band. Device 100 may also include or be coupled to a power source 118 for device 100, such as a battery or a component that couples device 100 to an energy source. Device 100 may also include or be coupled to Figure 1 Additional features or components not shown in the figure. In one example, a wireless interface that may include multiple transceivers and a baseband processor may be coupled to or included in the WAN adapter 152 for wireless communication devices. In another example, an analog front end (AFE) for converting analog image frame data into digital image frame data may be coupled between the image sensors 101 and 102 and the image signal processor 112.

[0045] The device may include or be coupled to a sensor hub 150 for interfacing with sensors to receive data about the movement of the device 100, data about the environment surrounding the device 100, and / or other non-camera sensor data. One example non-camera sensor is a gyroscope, i.e., a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, i.e., a device configured to measure acceleration, which may also be used to determine velocity and distance traveled by appropriately integrating the measured acceleration, and one or more of the acceleration, velocity, and / or distance may be included in the generated motion data. In some aspects, a gyroscope in an electronic image stabilization system (EIS) may be coupled to the sensor hub or directly to the image signal processor 112. In another example, the non-camera sensor may be a global positioning system (GPS) receiver.

[0046] The image signal processor 112 may receive image data such as for forming an image frame. In one embodiment, a local bus connection couples the image signal processor 112 to the image sensors 101 and 102 of the first camera 103 and the second camera 105, respectively. In another embodiment, a wire interface couples the image signal processor 112 to an external image sensor. In yet another embodiment, a wireless interface couples the image signal processor 112 to the image sensors 101, 102.

[0047] The first camera 103 may include a first image sensor 101 and a corresponding first lens 131. The second camera may include a second image sensor 102 and a corresponding second lens 132. Each of the lenses 131 and 132 may be controlled by an associated auto focus (AF) algorithm 133 executed in the ISP 112, which adjusts the lenses 131 and 132 to focus on a specific focal plane at a certain scene depth from the image sensors 101 and 102. The AF algorithm 133 may be assisted by a depth sensor 140.

[0048] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. The lenses 131 and 132 focus light at the image sensors 101 and 102, respectively, through one or more apertures for receiving light, one or more shutters for blocking light when outside the exposure window, one or more color filter arrays (CFA) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurements into digital information, and / or other suitable components for imaging. The first lens 131 and the second lens 132 may have different fields of view to capture different representations of the scene. For example, the first lens 131 may be an ultra-wide (UW) lens, and the second lens 132 may be a wide (W) lens. The plurality of image sensors may include a combination of ultra-wide (high field of view (FOV)) sensors, wide sensors, long-range sensors, and ultra-long-range (low FOV) sensors.

[0049] That is, each image sensor can be configured by hardware configuration and / or software settings to obtain a different but overlapping field of view. In one configuration, the image sensor is configured with different lenses with different magnifications, which results in different fields of view. The sensors can be configured so that the UW sensor has a larger FOV than the W sensor, the W sensor has a larger FOV than the T sensor, and the T sensor has a larger FOV than the UT sensor. For example, a sensor configured for a wide FOV can capture a field of view in the range of 64 degrees to 84 degrees, a sensor configured for an ultra-lateral FOV can capture a field of view in the range of 100 degrees to 140 degrees, a sensor configured for a long-range FOV can capture a field of view in the range of 10 degrees to 30 degrees, and a sensor configured for an ultra-long-range FOV can capture a field of view in the range of 1 degree to 8 degrees.

[0050] The image signal processor 112 processes the image frames captured by the image sensors 101 and 102. Figure 1The device 100 is illustrated as including two image sensors 101 and 102 coupled to an image signal processor 112, but any number (e.g., one, two, three, four, five, six, etc.) of image sensors may be coupled to the image signal processor 112. In some aspects, a depth sensor such as a depth sensor 140 may be coupled to the image signal processor 112, and the output from the depth sensor may be processed in a manner similar to that of the image sensors 101 and 102. Example depth sensors include active sensors including one or more of indirect time of flight (iToF), direct time of flight (dToF), light detection and ranging (lidar), mmWave, radio detection and ranging (radar), and / or hybrid depth sensors (such as structured light). In embodiments without the depth sensor 140, similar information about the depth or depth map of an object can be generated in a passive manner from the parallax between the two image sensors (e.g., using parallax depth measurement or stereo depth measurement), a phase detection autofocus (PDAF) sensor, etc. In addition, there may be any number of additional image sensors or image signal processors for the device 100.

[0051] In some embodiments, the image signal processor 112 may execute instructions from a memory, such as instructions 108 from the memory 106, instructions stored in a separate memory coupled to or included in the image signal processor 112, or instructions provided by the processor 104. In addition or in the alternative, the image signal processor 112 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described in the present disclosure. For example, the image signal processor 112 may include one or more image front ends (IFEs) 135, one or more image post-processing engines 136 (IPEs), and / or one or more automatic exposure compensation (AEC) 134 engines. The AF 133, AEC 134, IFE 135, IPE 136 may each include dedicated circuitry embodied as software code executed by the ISP 112 and / or a combination of hardware within the ISP 112 and software code executed on the ISP 112.

[0052] In some implementations, the memory 106 may include a non-transitory or non-transitory computer-readable medium storing computer-executable instructions 108 to perform all or part of one or more operations described in the present disclosure. In some implementations, the instructions 108 include a camera application (or other suitable application) to be executed by the device 100 for generating images or videos. The instructions 108 may also include other applications or programs executed by the device 100, such as an operating system and specific applications other than for image or video generation. A camera application such as that executed by the processor 104 may enable the device 100 to generate images using the image sensors 101 and 102 and the image signal processor 112. The memory 106 may also be accessed by the image signal processor 112 to store processed frames, or may be accessed by the processor 104 to obtain processed frames. In some embodiments, the device 100 does not include the memory 106. For example, the device 100 may be a circuit including the image signal processor 112, and the memory may be external to the device 100. The device 100 may be coupled to an external memory and configured to access the memory to write output frames for display or long-term storage. In some embodiments, device 100 is a system on a chip (SoC) that combines image signal processor 112, processor 104, sensor hub 150, memory 106, and input / output components 116 into a single package.

[0053] In some embodiments, at least one of the image signal processor 112 or the processor 104 executes instructions to perform various operations described herein, including variable aperture (VA) control and image processing operations. For example, execution of the instructions may instruct the image signal processor 112 to start or end capturing an image frame or a sequence of image frames, wherein the capture includes variable aperture (VA) control and image processing as described in the embodiments herein. In some embodiments, the processor 104 may include one or more general-purpose processor cores 104A capable of executing scripts or instructions of one or more software programs (such as instructions 108 stored in the memory 106). For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in the memory 106.

[0054] When executing the camera application, the processor 104 can be configured to command the image signal processor 112 to perform one or more operations with reference to the image sensor 101 or 102. For example, the camera application executed on the processor 104 may receive a user command to start a video preview display, and upon receiving the user command, a video including a sequence of image frames is captured and processed from one or more image sensors 101 or 102 by the image signal processor 112. Image processing such as that used to produce an "output" or "corrected" image frame according to the techniques described herein may be applied to one or more image frames in the sequence. Execution of instructions 108 by the processor 104 outside the camera application can also cause the device 100 to perform any number of functions or operations. In some embodiments, the processor 104 may include an IC or other hardware (e.g., an artificial intelligence (AI) engine 124 or other coprocessor) to offload certain tasks from the core 104A. The AI ​​engine 124 can be used to offload tasks related to, for example, facial detection and / or object recognition. In some other embodiments, the device 100 does not include the processor 104, such as when all of the described functionality is configured in the image signal processor 112.

[0055] In some embodiments, display 114 may include one or more suitable displays or screens that allow user interaction and / or present items to the user (such as previews of image frames captured by image sensors 101 and 102). In some embodiments, display 114 is a touch-sensitive display. I / O component 116 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from a user and provide output to the user through display 114. For example, I / O component 116 may include (but is not limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (e.g., a power button), a slider, a switch, etc.

[0056] Although shown coupled to each other via processor 104, components such as processor 104, memory 106, image signal processor 112, display 114, and I / O components 116 may be coupled to each other in various other arrangements, such as via one or more local buses, which are not shown for simplicity. Although image signal processor 112 is illustrated as being separate from processor 104, image signal processor 112 may be a core of processor 104, which is an application processor unit (APU), included in a system on chip (SoC), or otherwise included in processor 104. Although reference is made to device 100 in the examples herein to perform aspects of the present disclosure, some device components may not be present. Figure 1100. In some embodiments, the present invention is not limited to the configuration of a particular device or component, including device 100.

[0057] Actionable Figure 1 An exemplary image capture device of the present invention can obtain improved images by controlling the configuration of one or more cameras, such as camera 103, to obtain image frames of a photo or video with multiple objects in focus. Control of camera 103 may include adjusting the aperture size of a variable aperture in one or more cameras and / or adjusting the lens position to set the depth of focus of camera 103. Figure 2 One example method of operating one or more cameras, such as camera 103, is shown in FIG. 4 and described below.

[0058] Figure 2 2 is a block diagram illustrating an example data flow path for image data processing in an image capture device according to one or more embodiments of the present disclosure. The processor 104 of the system 200 can communicate with the image signal processor (ISP) 112 via a bidirectional bus and / or separate control and data lines. The processor 104 can control the camera 103 via a camera control 210, such as for configuring the camera 103 through a driver executed on the processor 104. The camera control 210 can be managed by a camera application 204 executed on the processor 104, which provides user-accessible settings so that the user can specify individual camera settings or select a profile with corresponding camera settings. The camera control 210 communicates with the camera 103 to configure the camera 103 according to commands received from the camera application 204. The camera application 204 can be, for example, a photography application, a document scanning application, a messaging application, or other application that processes image data acquired from the camera 103.

[0059] The camera configuration may be parameters that specify, for example, frame rate, image resolution, readout duration, exposure level, aspect ratio, aperture size, and the like. Figure 3 An example effect of aperture size on the output image frame is shown in FIG. Figure 3306. FIG. 306 is a graphical representation of a variable aperture at several aperture sizes that may be controlled by a camera configuration according to one or more embodiments of the present disclosure. The variable aperture may be configured to correspond to one of the configurations of aperture 302, aperture 304, or aperture 306. A smaller aperture value, such as an aperture value (or "f-stop") f / 2.8 for aperture 302, results in a larger opening, thereby increasing the amount of light reaching the image sensor in the shadow of aperture 302. A larger aperture value, such as an aperture value (or "f-stop") f / 22 for aperture 306, results in a smaller opening, thereby reducing the amount of light reaching the image sensor in the shadow of aperture 306. Other aperture sizes, such as an aperture value (or "f-stop") f / 5.6 for aperture 304, provide additional variability in the characteristics of the captured image frames. The effect of the various aperture sizes 302, 304, and 306 may be to brighten or darken the image frames captured through the apertures. Another effect of the various aperture sizes 302, 304, and 306 may be to adjust the depth of focus of the image frame captured by the aperture. A smaller aperture value may result in an image frame having a lower depth of field (as shown in image 312), thereby increasing blur in the image frame at depths far from the depth of focus. A larger aperture value may result in an image frame having a higher depth of field (as shown in image 314), thereby reducing blur in the image frame at depths far from the depth of focus.

[0060] Reference again Figure 2 , the camera 103 may obtain image data based on the camera configuration. For example, the processor 104 may execute the camera application 204 to instruct the camera 103 to set a first image sensor configuration of the camera 103 through the camera control 210, obtain first image data from the camera 103 operating with the first camera configuration, instruct the camera 103 to set a second camera configuration of the camera 103, and obtain second image data from the camera 103 operating with the second camera configuration.

[0061] In an example operation, the camera application 204 may determine that the presence of one or more regions in a scene within the field of view of the camera 103 is in focus within the image frame. The one or more regions may be determined based on user input, such as a user tapping a point on the screen or drawing an area on the screen. The one or more regions may be determined based on an algorithm, such as an algorithm that identifies bright areas of the scene. The one or more regions may be determined based on an analysis of the scene, such as by using an object detector (e.g., a face detector) to identify areas of the scene with objects of interest.

[0062] Image data received from camera 103 may be processed in one or more blocks of ISP 112 to form image frames 230 that are stored in memory 106 and / or provided to processor 104. Processor 104 may further process the image data to apply effects to image frames 230. Effects may include Bokeh, lighting, color cast, and / or high dynamic range (HDR) merging. In some embodiments, functionality may be embedded in different components, such as ISP 112, DSP, ASIC, or other custom logic circuitry for performing additional image processing.

[0063] Figure 2 The system 200 may be configured to perform reference Figure 4 The operations described are to determine the output image frame 230 . Figure 4 A flowchart is shown of an example method for controlling a camera to capture an image with a desired portion of a scene in focus, according to some embodiments of the present disclosure. Figure 4 Capturing in the image may result in an improved digital representation of the scene, resulting in a photo or video with higher image quality (IQ).

[0064] At block 402, a plurality of depths may be determined for a scene, wherein each of the plurality of depths corresponds to an area of ​​interest in the scene. The areas of interest in the scene may be determined by user input or by an algorithm executed on an image capture device. Each area of ​​interest may have a different depth or be at a different distance from the image capture device, such that certain aperture sizes may cause some areas of interest to be out of focus. This may result in capturing a representation of the scene that is not desired by the user because the areas of interest are out of focus and blurry, resulting in loss of detail in the areas of interest. For example, two people may be located at different distances from the image capture device, one of whom is in the foreground of the scene and the other in the near background of the scene. When one person is out of focus, the user may not be able to determine the identity of the person. For another example, two objects at a historical location may be located at different distances from the image capture device, one of whom is in the foreground of the scene and the other in the near background of the scene. The user may desire that both objects in the photograph are in focus. Different areas of interest (e.g., detected persons and / or detected objects) may be identified and depths corresponding to the areas may be determined. For example, the depth may be determined using data captured by a depth sensor from the scene. In some embodiments, multiple regions of interest and multiple depths may be determined based on one or more of previous image data captured from a camera of an image capture device and / or a depth sensor of an image capture device. In some embodiments, a saliency-based approach may be used to determine multiple regions of interest. Saliency refers to the likelihood that a human viewer will focus on a portion of a scene when the viewer is only instructed to view the scene. In embodiments employing a saliency model, scores may be assigned to pixels or portions of a scene, and portions of the scene for which saliency scores exceed a threshold may be determined as regions of interest. In some embodiments, a high saliency score may indicate the presence of an object in a scene, or in some embodiments, the presence of a face in a scene.

[0065] A camera configuration for capturing image data may be determined based on the multiple depths determined at block 402. At block 404, an aperture size of a camera for capturing image data may be determined based on the multiple depths. For example, a difference between one of the multiple depths (e.g., a maximum depth) and a second of the multiple depths (e.g., a minimum depth) may be determined as a desired depth of focus (DOF) for the scene. The aperture size may be selected to obtain a depth of focus at least as large as the desired depth of focus. In some embodiments, the depth of focus may take precedence over other characteristics of the camera such that the aperture size is not increased beyond an aperture size that satisfies the desired depth of focus. In some embodiments, the depth of focus may be prioritized in conjunction with other characteristics of the camera such that a balance is achieved between the depth of focus and, for example, scene brightness. In this embodiment, the determined aperture size may be determined within a range around the desired depth of focus, where the range may be a predetermined percentage or number of "f-stops" around the desired depth of focus.

[0066] Some examples of priorities are provided below. In some embodiments of a variable aperture, the available aperture sizes may be discrete "f-stops" such as f / 1.2, f / 1.4, f / 1.8, f / 2.0, f / 2.2, f / 2.4, f / 2.8, f / 3.2, f / 4.0, etc. When the desired depth of focus corresponds to an aperture size between f / 1.2 and f / 1.4 and the camera control prioritizes aperture size, the camera control may determine an aperture size of f / 1.2, which is the smallest aperture that satisfies the desired depth of focus. When the desired depth of focus corresponds to an aperture size between f / 1.2 and f / 1.4 and the camera control prioritizes brightness control to obtain a darker image, the camera control may determine an aperture size of f / 1.4. In some scenes, an aperture size with a corresponding depth of field that is within a predetermined amount of the desired depth of field may be acceptable. In some embodiments of a variable aperture, the available aperture sizes may be continuous, and an equation or artificial intelligence (AI) may be used to determine the aperture size that satisfies one or more conditions, wherein at least one condition corresponds to a desired depth of focus. In some embodiments, the aperture size may be determined by determining an initial aperture size based on light conditions in a scene; and adjusting the initial aperture size to determine the aperture size based on multiple depths.

[0067] At block 406, a lens position of a camera for capturing image data may be determined based on the multiple depths. The lens position sets a focus for the camera that centers the depth of focus for the aperture size. That is, the range of depths in focus in the scene is set around a depth corresponding to the depth of focus at the focus determined by the lens position. The lens position may be determined so that the depth of focus for the aperture size determined at block 404 brings the region of interest in the scene into focus. The lens position may be determined based on one or more priorities. In some embodiments, such as in some examples described with reference to block 404, the determined aperture size may have a depth of focus that is less than the desired depth of focus. The lens position may be determined at block 406 to place the depth of focus between two regions of interest, thereby obtaining a scene representation that has as many regions in focus as possible for the aperture size. The lens position may be determined in block 406 to prioritize one region of interest, such that the selected lens position places the prioritized region of interest within the depth of focus, but at the expense of reducing the sharpness of other identified regions of interest.

[0068] In some implementations, when the determined aperture size and lens position partially blur a portion of the region of interest, a deblurring operation may be applied to at least a portion of one of the plurality of regions of interest that exceeds the depth of focus of the camera. The deblurring operation may improve sharpness in the region of interest to obtain a scene representation that obtains a synthetic depth of focus that is greater than the actual depth of focus for the aperture size and lens position.

[0069] At block 408, the camera is controlled to adjust the variable aperture (VA) of the camera based on the aperture size determined at block 404 and to adjust the lens of the camera based on the lens position determined at block 406. The camera configuration may be (e.g., with reference to Figure 2 ) is determined by the camera application 204, provided to the camera control 210, and sent from the processor 104 to the camera 103 via a bus or other interface. The first image data may be received from the camera 103, such as when the image sensor is configured with the camera configuration. The first image data may be received at the ISP 112, processed by the image front end (IFE) and / or image post-processing engine (IPE) of the ISP 112, and stored in a memory. In some embodiments, the capture of the image data may be initiated by a camera application executed on the processor 104, which causes the camera control 210 to activate the capture of the image data by the camera 103, and causes the image data to be provided to a processor, such as the processor 104 or the ISP 112.

[0070] Reference again Figure 4 At block 410, an output image frame is determined based on the first image data captured using the camera configuration set in the camera at block 408. Figure 2 In the example of , image frame 230 may be determined by processor 104 or ISP 112 and stored in memory 106. The stored image frame may be read by processor 104 and used to form a preview display on a display of device 100 (such as by camera application 204) and / or processed to form a photo to be stored in memory 106 and / or sent to another device.

[0071] Figure 5 1 is a block diagram illustrating an example processor configuration for capturing image data using a variable aperture (VA) camera according to one or more embodiments of the present disclosure. Processor 104 may receive first image data (such as previously captured data from camera 103 or other cameras) and determine characteristics of a scene therefrom, including the presence of one or more objects. Processor 104 may receive second image data (such as depth sensor data) and generate a depth map therefrom. In some embodiments, the second image data may be additional image data from another image sensor, such that a depth map may be generated from a disparity calculated from the first image data and the second image data.

[0072] The processor 104 or other processing circuitry may be configured to perform Figure 4One or more operations of the method of controlling one or more cameras to capture image data having a desired scene representation having sharpness in a plurality of regions of interest. At block 410, the image data may be processed to determine one or more output image frames. Figure 6 The exemplary embodiments shown are described Figure 5 The operation of the processor 104 in. Figure 6 is a graphical representation of camera control of a variable aperture (VA) for an example scene, in accordance with one or more embodiments of the present disclosure.

[0073] The processor 104 executes the depth map generator 504A to determine the depth map. Figure 6 An example depth 602 is shown. The depth map may divide a scene into a plurality of blocks and assign a depth value to each block. Each block may represent a plurality of pixels of a scene representation in the image data. In this configuration, the depth value of the block may be determined based on a defined function, such as by determining the depth of the block based on the maximum depth of any pixel in the block, by determining the depth of the block based on the minimum depth of any pixel in the block, by determining the depth of the block based on the median value of the pixels in the block, or by determining the depth of the block based on the average value of the pixels in the block. The values ​​of the depth map may indicate an actual determined depth, such as a value named in units of distance (e.g., centimeters or inches). The values ​​of the depth map may alternatively indicate a relative depth, which may or may not be named in units of distance. For example, the depth map 602 may include a disparity value.

[0074] The processor 104 executes a depth of field (DOF) estimator 504B to determine a plurality of depths corresponding to a plurality of regions of interest for a scene. The DOF estimator 504B may identify objects (such as people) in the scene and corresponding regions of interest 612, 614 in the scene. Object recognition (such as facial recognition) may be performed using, for example, the AI ​​engine 124 of the processor 104. A depth value may be determined for each of the regions of interest 612, 614 to associate the depth value with the object. In the example depth map 602, a depth of 50 centimeters may be associated with the region 612 and a depth of 130 centimeters may be associated with the region 614.

[0075] The processor 104 executes an aperture selector 504C to determine an aperture size based on the desired depth of focus determined by the DOF estimator 504B. The aperture size may be selected from the available aperture sizes (e.g., f / 1.4, f / 2.8, f / 4.0) based on the depth of focus of each available aperture size and the desired depth of focus. For example, a first aperture size of f / 1.4 may have a DOF 622, a second aperture size of f / 2.8 may have a DOF 624, and a third aperture size of f / 4.0 may have a DOF 626. In an example scene with two regions of interest corresponding to two people, the DOF 622 and the DOF 624 may not be sufficient to provide a clear representation of the two people. The DOF 626 may be large enough to provide a clear representation of the two people. In this example, the aperture selector 504C may select the aperture size of f / 4.0, but the criteria for selecting the aperture may not only be the desired depth of focus. Processor 104 may include the selected aperture size, or an indication of the selected aperture size, in a camera configuration sent to camera 103 .

[0076] The processor 104 executes the focus depth estimator 504D to determine the focus depth and the corresponding lens position to achieve the focus corresponding to the focus depth. The focus depth can center the depth of focus area of ​​the selected aperture. Adjusting the focus depth can adjust the depth area that appears sharp at the selected aperture size from the aperture selector 504C. For example, the first focus depth 632 can place the first object within the depth of focus of a specific aperture size. Similarly, the second focus depth 632 can place the first object within the depth of focus of a specific aperture size, but is still not enough to place the second object within the depth of focus. The third focus depth 636 can place the first object and the second object within the depth of focus of the specific aperture size. The focus depth estimator 504D can select the focus depth 636. The lens position in the camera can be determined for the focus depth 636. In some embodiments, the processor 104 sends a camera configuration including an indication of the focus depth 636 to the camera 103. In some embodiments, processor 104 sends a camera configuration including a lens position corresponding to focus depth 636 to camera 103 .

[0077] The camera control 210 uses the outputs of the aperture selector 504C and the focus depth estimator 504D to determine the camera configuration. The camera configuration is sent to the camera 103 to control various aspects of the first lens 131, the variable aperture 510, and / or the first image sensor 101.

[0078] In some embodiments, the Figure 7 The focus depth estimator 504D of the example is used to estimate the focus depth. Figure 7is a graphical representation of lens position control operations for a variable aperture (VA) camera according to one or more embodiments of the present disclosure. A focus depth estimator 504D may first retrieve the depth of each region of interest or object in the region of interest from, for example, a lookup table (LUT). The LUT may be a table of extended depths of focus associated with distances. The result may locate each object along a linear dimension corresponding to the depth of the image sensor, where each object has a depth of field (DOF) around its position in the linear dimension. The DOF may correspond to the DOF at a determined aperture size. The focus depth estimator 504D may use the list of objects along the depth dimension and their associated DOF to determine a lens position such that a maximum number of objects are in focus at the determined aperture size and lens position.

[0079] The described technique for controlling a camera of an image capture device can be used to obtain a photo with multiple objects in focus. The variable aperture (VA) of the camera can be controlled to obtain sharp objects in multiple regions of interest, thereby obtaining improved photography results. Figure 8 Examples of such photographs are shown. Figure 8 800 is a graphical representation of image data captured from a variable aperture (VA) camera according to one or more embodiments of the present disclosure. Photo 800 may include a first object 802 and a second object 804 captured in sharp detail and a third object 806 in a blurred background. Because the variable aperture (VA) of the camera is controlled to provide an aperture size that covers the depth of focus of both objects 802 and 804, and the lens position of the camera is controlled to obtain a focus depth that covers the depth of focus of both objects 802 and 804, the first and second objects 802 and 804 are captured in a focused state.

[0080] In one or more aspects, techniques for supporting image processing may include additional aspects, such as any single aspect or any combination of aspects described below or in conjunction with one or more other processes or devices described elsewhere herein. In a first aspect, supporting image processing may include an apparatus configured to: determine multiple depths for a scene, the multiple depths corresponding to multiple regions of interest; determine an aperture size of a camera based on the multiple depths; and / or determine a lens position of the camera based on the aperture size and the multiple depths. The apparatus according to the first aspect may be further configured to control the camera to adjust a variable aperture (VA) of the camera based on the aperture size and adjust a lens associated with the variable aperture (VA) based on the lens position.

[0081] Additionally, the device may be performed or operated according to one or more aspects as described below. In some specific implementations, the device includes a wireless device, such as a UE. In some specific implementations, the device includes a remote server (such as a cloud-based computing solution) that receives image data for processing to determine an output image frame. In some specific implementations, the device may include at least one processor, and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the device. In some other specific implementations, the device may include a non-transitory computer-readable medium having a program code recorded thereon, and the program code may be executable by a computer for causing the computer to perform the operations described herein with reference to the device. In some specific implementations, the device may include one or more components configured to perform the operations described herein. In some specific implementations, the method of wireless communication may include one or more operations described herein with reference to the device.

[0082] In a second aspect, in combination with the first aspect, the apparatus is further configured to determine a plurality of objects of the scene, wherein the plurality of depths correspond to the plurality of objects.

[0083] In a third aspect, in combination with one or more of the first aspect or the second aspect, determining the aperture size includes determining an aperture size having a focal depth at least as great as a distance between a first depth (e.g., a minimum depth) of the multiple depths and a second depth (e.g., a maximum depth) of the multiple depths.

[0084] In a fourth aspect, in combination with one or more of the first to third aspects, determining the lens position includes determining the lens position so that the focal depth around the lens position includes at least a minimum depth among the multiple depths and a maximum depth among the multiple depths.

[0085] In a fifth aspect, in combination with one or more of the first to fourth aspects, determining multiple depths for a scene includes determining positions of at least two faces in the scene, wherein the multiple depths correspond to depths at the positions of the at least two faces.

[0086] In a sixth aspect, in combination with one or more of the first to fifth aspects, determining the aperture size of the camera includes: determining an initial aperture size based on light conditions in the scene; and adjusting the initial aperture size to determine the aperture size based on multiple depths.

[0087] In a seventh aspect, in combination with one or more of the first to sixth aspects, determining a plurality of depths for a scene includes receiving depth data of the scene from a depth sensor.

[0088] In an eighth aspect, in combination with one or more of aspects one to seven, depth is obtained from one or more of active depth sensors (e.g., indirect time of flight (iToF), direct time of flight (dToF), light detection and ranging (lidar), mmWave, radio detection and ranging (radar)), hybrid depth sensors (such as structured light), and / or passive depth sensing (e.g., parallax between two image sensors (e.g., using parallax depth measurement or stereo depth measurement) and / or phase detection autofocus (PDAF) sensors).

[0089] In a ninth aspect, in combination with one or more of the first to eighth aspects, image data is received from a camera, the image data being captured with the variable aperture and at the lens position; and at least one image frame is determined based on the image data.

[0090] In the tenth aspect, in combination with one or more of the first to ninth aspects, the device is further configured to determine a distance between a camera focal depth at the lens position and with the aperture size that is less than a first depth (e.g., a minimum depth) among the multiple depths and a second depth (e.g., a maximum depth) among the multiple depths, wherein determining the at least one image frame includes deblurring at least a portion of an area of ​​interest among the multiple areas of interest that exceeds the camera focal depth.

[0091] In an eleventh aspect, in combination with one or more of the first to tenth aspects, the apparatus is further configured to determine positions of at least two faces in the scene.

[0092] In a twelfth aspect, in combination with one or more of the first to eleventh aspects, a user input is received specifying a plurality of regions of interest from positions of at least two faces in a scene.

[0093] In the thirteenth aspect, in combination with one or more of the first to twelfth aspects, the multiple depths correspond to a face selected by a user in the scene, determining the aperture size includes determining the aperture size with a focal depth based on a distance between a first depth (e.g., a minimum depth) among the multiple depths and a second depth (e.g., a maximum depth) among the multiple depths, and determining the lens position includes determining the lens position based on a distance between the minimum depth among the multiple depths and the maximum depth among the multiple depths.

[0094] In a fourteenth aspect, in combination with one or more of the first to thirteenth aspects, the apparatus is further configured to receive image data from a camera, the image data being captured with the variable aperture and comprising; and determining at least one image frame based on the image data.

[0095] In a fifteenth aspect, in combination with one or more of aspects one to fourteen, the device may include a camera comprising an image sensor, a variable aperture (VA), and a positionable lens (such as having mechanical, electromechanical, or electrical control for adjusting the distance between the lens and the image sensor).

[0096] In the sixteenth aspect, in combination with one or more of the first to fifteenth aspects, the device is further configured to receive first image data representing a scene from a camera and to determine multiple positions of multiple objects of the scene by performing object detection on the first image data, wherein the multiple depths correspond to the multiple objects.

[0097] In the seventeenth aspect, in combination with one or more of the first to sixteenth aspects, the device is further configured to receive user input specifying the multiple regions of interest from the multiple positions, wherein the multiple depths correspond to objects selected by the user in the scene, wherein determining the aperture size includes determining the aperture size with a focal depth based on a distance between a first depth in the multiple depths and a second depth in the multiple depths, and wherein determining the lens position includes determining the lens position based on a distance between a first depth in the multiple depths and a second depth in the multiple depths.

[0098] In an eighteenth aspect, in combination with one or more of the first to seventeenth aspects, determining multiple positions of multiple objects of the scene includes determining multiple positions of multiple faces of the scene by performing facial recognition on the first image data.

[0099] In a nineteenth aspect, in combination with one or more of the first to eighteenth aspects, determining multiple locations of multiple objects of a scene includes determining the multiple locations by applying a saliency model to the first image data to determine locations having saliency scores above a threshold.

[0100] In the twentieth aspect, in combination with one or more of the first to nineteenth aspects, the device is further configured to receive image data from a camera, the image data being captured based on the aperture size and the lens position (such as when a seed for an autofocus algorithm is the determined lens position, and the autofocus algorithm further refines precise focus or refocuses to track an object in an area of ​​interest, performing operations based on the lens position); and determining at least one image frame based on the image data.

[0101] It should be understood by those skilled in the art that any of a variety of different technologies and techniques may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0102] In this article Figures 1 to 6 The components, functional blocks and modules described include processors, electronic devices, hardware devices, electronic components, logical circuits, memories, software codes, firmware codes, etc. or any combination thereof. Software should be broadly interpreted to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, and / or functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language or other terms. In addition, the features discussed herein may be implemented via dedicated processor circuits, via executable instructions, or a combination thereof.

[0103] It should be understood by those skilled in the art that: Figure 4 and Figure 5 One or more blocks (or operations) described herein may be combined with one or more blocks (or operations) described in another figure in the reference figure. Figure 4 One or more boxes (or operations) of Figure 1 One or more boxes (or operations) of . Figure 5 One or more boxes associated with Figure 1 One or more associated boxes (or operations) are combined. Additionally or alternatively, the above reference Figure 1 to Figure 2 One or more of the operations described may be combined with Figures 3 to 5 A combination of one or more of the operations described.

[0104] It should also be appreciated by those of ordinary skill in the art that the various illustrative logic boxes, modules, circuits, and algorithmic steps described in conjunction with the disclosure herein can all be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. The technician will also easily recognize that the order or combination of components, methods, or interactions described herein are merely examples, and the components, methods, or interactions of various aspects of the present disclosure can be combined or performed in a manner other than the manner illustrated and described herein.

[0105] The various illustrative logical components, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. The interchangeability of hardware and software has been generally described in terms of functionality and is illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0106] The hardware and data processing devices for implementing the various exemplary logic components, logic blocks, modules and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some specific implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. In some specific implementations, a particular process and method may be performed by a circuit specific to a given function.

[0107] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, including the structures disclosed in this specification and their structural equivalents, or any combination thereof. Specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on computer storage media for execution by data processing apparatus or for controlling the operation of data processing apparatus.

[0108] If implemented in software, the function may be stored as one or more instructions or codes on a computer-readable medium or sent via a computer-readable medium. The process of the method or algorithm disclosed herein may be implemented in a processor executable software module that may reside on a computer-readable medium. Computer-readable media include both computer storage media and communication media, and the communication media include any media that can be implemented to transfer a computer program from one place to another. The storage medium may be any available medium that a computer can access. As an example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the required program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection may be appropriately referred to as a computer-readable medium. Disks and optical disks as used herein include compact discs (CDs), laser discs, optical disks, digital versatile discs (DVDs), floppy disks, and blue discs, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as a code and instruction set, or any combination of code and instruction sets, on a machine-readable medium or computer-readable medium, which may be incorporated into a computer program product.

[0109] Various modifications to the specific implementations described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to some other specific implementations without departing from the spirit or scope of the disclosure. Therefore, the claims are not intended to be limited to the specific implementations shown herein, but are to be consistent with the broadest scope consistent with the disclosure, the principles and novel features disclosed herein.

[0110] In addition, one of ordinary skill in the art will readily recognize that, for ease of describing the drawings, opposing terms such as "upper" and "lower" or "front" and "back" or "top" and "bottom" are sometimes used and indicate relative positions corresponding to the orientation of the drawings on a correctly oriented page, and may not reflect the correct orientation of any device as implemented.

[0111] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be removed from the combination, and a claimed combination may be directed to subcombinations or variations of subcombinations.

[0112] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all illustrated operations to achieve the desired result. In addition, the accompanying drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations that are not depicted may be combined in the schematically illustrated example process. For example, one or more additional operations may be performed before, after, simultaneously or between any illustrated operations. In some environments, multitasking and parallel processing are advantageous. In addition, the separation of various system components in the specific implementation described above should not be understood as requiring such separation in all specific implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other specific implementations also fall within the scope of the appended claims. In some cases, the actions narrated in the claims may be performed in different orders and still achieve the desired result.

[0113] As used herein, including in the claims, the term "or" used in a list of two or more items means that any of the listed items may be employed alone, or any combination of two or more of the listed items may be employed. For example, where a composition is described as containing components A, B, or C, the composition may contain A alone; B alone; C alone; A and B in combination; A and C in combination; B and C in combination; or A, B, and C in combination. In addition, as used herein, including in the claims, "or" as used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination of any of these. The term "substantially" is defined as to a large extent, but not necessarily entirely, what is specified (and includes what is specified; for example, substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by one of ordinary skill in the art. In any disclosed implementation, the term "substantially" may be replaced with "within [percent] of" a specified content, where the percentage includes 0.1%, 1%, 5%, or 10%.

[0114] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Therefore, the disclosure is not intended to be limited to the examples and designs described herein, but to be consistent with the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method, comprising: determining a plurality of depths for a scene, the plurality of depths corresponding to a plurality of regions of interest; determining an aperture size of a camera based on the plurality of depths; determining a lens position of the camera based on the aperture size and the plurality of depths; as well as The camera is controlled to adjust a variable aperture (VA) of the camera based on the aperture size and a lens associated with the variable aperture (VA) is adjusted based on the lens position. 2 . The method of claim 1 , wherein determining the aperture size comprises determining an aperture size having a depth of focus at least as great as a distance between a first depth in the plurality of depths and a second depth in the plurality of depths.

3. The method of claim 2, wherein determining the lens position comprises determining a lens position such that the depth of focus around the lens position includes at least the first depth of the plurality of depths and the second depth of the plurality of depths. 4 . The method of claim 1 , wherein determining the plurality of depths for a scene comprises determining locations of at least two faces in the scene, wherein the plurality of depths corresponds to depths at the locations of the at least two faces.

5. The method of claim 1 , wherein determining the aperture size of the camera comprises: determining an initial aperture size based on light conditions in the scene; as well as The initial aperture size is adjusted to determine the aperture size based on the plurality of depths. The method of claim 1 , wherein determining the plurality of depths for a scene comprises receiving depth data for the scene from a depth sensor.

7. The method according to claim 1, further comprising: receiving image data from the camera, the image data being captured with the variable aperture and at the lens position; as well as At least one image frame is determined based on the image data.

8. The method according to claim 7, further comprising: determining that a camera focal depth of the camera at the lens position and at the aperture size is less than a distance between a first depth of the plurality of depths and a second depth of the plurality of depths, Wherein determining the at least one image frame comprises deblurring at least a portion of a region of interest among the plurality of regions of interest that is beyond the focal depth of the camera.

9. The method according to claim 1, further comprising: receiving first image data representing the scene from the camera; as well as A plurality of positions of a plurality of objects of the scene are determined by performing object detection on the first image data, wherein the plurality of depths correspond to the plurality of objects.

10. The method according to claim 9, further comprising: receiving a user input for specifying the plurality of regions of interest from the plurality of locations, wherein: The plurality of depths correspond to objects in the scene selected by a user; Determining the aperture size includes determining an aperture size having a focal depth based on a distance between a first depth in the plurality of depths and a second depth in the plurality of depths; and Determining a lens position includes determining a lens position based on a distance between the first depth of the plurality of depths and the second depth of the plurality of depths.

11. The method of claim 10, wherein determining the plurality of positions of a plurality of objects of the scene comprises determining a plurality of positions of a plurality of faces of the scene by performing facial recognition on the first image data.

12. The method of claim 10, wherein determining the plurality of locations of the plurality of objects of the scene comprises determining the plurality of locations by applying a saliency model to the first image data to determine locations having saliency scores above a threshold.

13. The method according to claim 10, further comprising: receiving second image data from the camera after controlling the camera to adjust a variable aperture (VA) of the camera based on the aperture size and adjusting a lens associated with the variable aperture (VA) based on the lens position, the second image data being captured based on the aperture size and the lens position; as well as At least one image frame is determined based on the second image data.

14. A device, comprising: a memory storing processor readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor readable code to cause the at least one processor to perform operations including: determining a plurality of depths for a scene, the plurality of depths corresponding to a plurality of regions of interest; determining an aperture size of a camera based on the plurality of depths; determining a lens position of the camera based on the aperture size and the plurality of depths; as well as The camera is controlled to adjust a variable aperture (VA) of the camera based on the aperture size and a lens associated with the variable aperture (VA) is adjusted based on the lens position.

15. The apparatus of claim 14, wherein determining the lens position comprises determining the lens position such that a depth of focus around the lens position with the aperture size includes at least a first depth of the plurality of depths and a second depth of the plurality of depths.

16. The apparatus of claim 15, wherein determining the lens position comprises determining a lens position such that the depth of focus around the lens position is at least as great as a distance between the first of the plurality of depths and the second of the plurality of depths.

17. The apparatus of claim 14, wherein determining the plurality of depths for a scene comprises determining locations of at least two faces in the scene, wherein the plurality of depths corresponds to depths at the locations of the at least two faces.

18. The apparatus of claim 14, wherein determining the aperture size of the camera comprises: determining an initial aperture size based on light conditions in the scene; as well as The initial aperture size is adjusted to determine the aperture size based on the plurality of depths.

19. The apparatus of claim 14, wherein determining the plurality of depths for a scene comprises receiving a depth map of the scene from a depth sensor.

20. The apparatus of claim 14, wherein the processor is further configured to perform operations comprising: receiving image data from the camera, the image data being captured with the variable aperture and at the lens position; and At least one image frame is determined based on the image data.

21. The apparatus of claim 20, wherein the processor is further configured to perform operations comprising: determining that a camera focal depth of the camera at the lens position and at the aperture size is less than a distance between a first depth of the plurality of depths and a second depth of the plurality of depths, Wherein determining the at least one image frame comprises deblurring at least a portion of a region of interest among the plurality of regions of interest that is beyond the focal depth of the camera.

22. The apparatus of claim 14, wherein the processor is further configured to perform operations comprising: receiving first image data representing the scene from the camera; and A plurality of positions of a plurality of objects of the scene are determined by performing object detection on the first image data, wherein the plurality of depths correspond to the plurality of objects.

23. The apparatus of claim 22, wherein the processor is further configured to perform operations comprising: receiving a user input for specifying the plurality of regions of interest from the plurality of locations, wherein: The plurality of depths correspond to objects in the scene selected by a user; Determining the aperture size includes determining an aperture size having a focal depth based on a distance between a first depth in the plurality of depths and a second depth in the plurality of depths; and Determining a lens position includes determining a lens position based on a distance between the first depth of the plurality of depths and the second depth of the plurality of depths.

24. The apparatus of claim 23, wherein determining the plurality of positions of a plurality of objects of the scene comprises determining a plurality of positions of a plurality of faces of the scene by performing facial recognition on the first image data.

25. The apparatus of claim 23, wherein determining the plurality of locations of the plurality of objects of the scene comprises determining the plurality of locations by applying a saliency model to the first image data to determine locations having saliency scores above a threshold.

26. An image capturing device, the image capturing device comprising: A camera comprising an image sensor, a variable aperture (VA), and a lens; a memory storing processor readable code; and at least one processor coupled to the memory and to the camera, the at least one processor configured to execute the processor readable code to cause the at least one processor to perform operations including: determining a plurality of depths for a scene, the plurality of depths corresponding to a plurality of regions of interest; determining an aperture size of the camera based on the multiple depths; determining a lens position of the camera based on the aperture size and the plurality of depths; as well as The camera is controlled to adjust the variable aperture (VA) of the camera based on the aperture size and the lens associated with the variable aperture (VA) is adjusted based on the lens position.

27. The image capture device of claim 26, wherein determining the aperture size comprises determining an aperture size having a depth of focus at least as great as a distance between a first depth in the plurality of depths and a second depth in the plurality of depths, and wherein determining a lens position comprises determining a lens position such that the depth of focus around the lens position includes at least the first depth in the plurality of depths and the second depth in the plurality of depths.

28. The image capture device of claim 27, wherein the processor readable code causes the at least one processor to perform further operations comprising: receiving first image data representing the scene from the camera; and A plurality of positions of a plurality of objects of the scene are determined by performing object detection on the first image data, wherein the plurality of depths correspond to the plurality of objects.

29. The image capture device of claim 28, wherein the processor readable code causes the at least one processor to perform further operations comprising: receiving a user input for specifying the plurality of regions of interest from the plurality of locations, wherein: The plurality of depths correspond to objects in the scene selected by a user; Determining the aperture size includes determining an aperture size having a focal depth based on a distance between a first depth in the plurality of depths and a second depth in the plurality of depths; and Determining a lens position includes determining a lens position based on a distance between the first depth of the plurality of depths and the second depth of the plurality of depths.

30. The image capture device of claim 29, wherein the processor readable code causes the at least one processor to perform further operations comprising: After controlling the camera to adjust the variable aperture (VA) of the camera based on the aperture size and adjusting the lens associated with the variable aperture (VA) based on the lens position, receiving second image data from the camera, the second image data being captured based on the aperture size and the lens position; and At least one image frame is determined based on the image data.

Images