Machine learning for phase detection autofocus

Abstract

This disclosure provides systems, methods, and devices for image capture using an auto-focus (AF) algorithm. In a first aspect, a method for autofocus includes: receiving first image data of a first scene from a first image sensor of a first camera, the first image data including phase shift information; determining a first focal length for the first scene by inputting the phase shift information to a machine learning algorithm based on the machine learning algorithm; and controlling a focal position of the first camera based on the first focal length. Other aspects and features are also claimed and described.

Description

Technical Field

[0001] Aspects of the present disclosure relate generally to image processing, and more particularly to autofocus systems for image capture devices. Some features enable and provide improved image processing, including the use of machine learning in determining the focal length of a scene. Background Art

[0002] An image capture device is a device that can capture one or more digital images (whether still images for photographs or image sequences for videos). Capture devices can be incorporated into various devices. By way of example, image capture devices may include stand-alone digital cameras or digital video cameras, wireless communication devices equipped with cameras, such as mobile phones, cellular or satellite radio phones, personal digital assistants (PDAs), tablet or tablet devices, gaming devices, computer devices (such as webcams, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0003] An auto focus (AF) algorithm improves the user experience by allowing the image capture device to be kept focused on an item of interest. When the AF algorithm fails to properly focus the image capture device, the image captured by the device appears blurry. A user viewing the blurry image recognizes the blur as reduced image quality. Photos or videos captured by an image capture device when the image capture device is not properly focused result in a reduced user experience, especially when the scene is changing rapidly and the image capture device cannot be reconfigured to obtain a second attempt at a photo of the scene. Summary of the invention

[0004] The following summarizes some aspects of the present disclosure to provide a basic understanding of the technology discussed. This summary is not an exhaustive overview of all expected features of the present disclosure, and is neither intended to identify key or important elements of all aspects of the present disclosure, nor to delineate the scope of any or all aspects of the present disclosure. The sole purpose of this summary is to present some concepts of one or more aspects of the present disclosure in a summarized form as a prelude to a more detailed description presented later.

[0005] An example autofocus (AF) algorithm is phase detection autofocus (PDAF), which operates using phase detection information obtained from an image sensor of an image capture device. With PDAF, a phase shift between left and right phase images generated by left and right phase detectors represents a focus level. When the AF algorithm controls a lens coupled to an image sensor for obtaining image data, the sign of the phase shift determines the direction for lens movement, and the magnitude of the phase shift determines the amount for moving the lens. If the lens is in a focused position, a small phase shift or zero phase shift may be determined when the left and right phase images are the same. If the lens is in an out-of-focus position, a large phase shift may be determined.

[0006] The PDAF operation for determining the phase shift in the AF algorithm can be based on machine learning to provide better performance in certain scene conditions when the AF algorithm has difficulty determining the phase shift. For example, in low light conditions, machine learning for phase detection (MLPD) can be used to determine the phase shift, and the machine learning-based phase shift value can be used to control the lens of the image capture device. In other examples, a repeating pattern or a moiré pattern can be detected, and MLPD can be used to determine the phase shift for controlling the lens. MLPD can be applied to other or all scene conditions as part of the AF algorithm executed on the image capture device. In some embodiments, MLPD can be selectively used as part of the AF algorithm by checking criteria to determine whether there are specific scene conditions that benefit from machine learning. In some embodiments, MLPD can configure a machine learning algorithm (such as a neural network) using different training weights based on the scene conditions to provide improved phase shift determination.

[0007] In one aspect of the present invention, a method for image processing (such as in an image capture device) includes: receiving first image data of a first scene from a first image sensor of a first camera, the first image data including phase shift information; determining a first focal length for the first scene based on a machine learning algorithm by inputting the phase shift information into the machine learning algorithm; and controlling a focus position of the first camera based on the first focal length.

[0008] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations including: receiving first image data of a first scene from a first image sensor of a first camera, the first image data including phase shift information; determining a first focal length for the first scene based on a machine learning algorithm by inputting the phase shift information into the machine learning algorithm; and controlling a focus position of the first camera based on the first focal length.

[0009] In an additional aspect of the present invention, an apparatus includes: a component for receiving first image data of a first scene from a first image sensor of a first camera, the first image data including phase shift information; a component for determining a first focal length for the first scene based on a machine learning algorithm by inputting the phase shift information into the machine learning algorithm; and a component for controlling a focus position of the first camera based on the first focal length.

[0010] In an additional aspect of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations including: receiving first image data of a first scene from a first image sensor of a first camera, the first image data including phase shift information; determining a first focal length for the first scene based on a machine learning algorithm by inputting the phase shift information into the machine learning algorithm; and controlling a focus position of the first camera based on the first focal length.

[0011] Image capture devices (devices that can capture one or more digital images, whether still images, photographs or video sequences of images) can be incorporated into a variety of devices. By way of example, image capture devices may include stand-alone digital cameras or digital video cameras, wireless communication devices equipped with cameras, such as mobile phones, cellular or satellite radio phones, personal digital assistants (PDAs), tablet or tablet devices, gaming devices, computer devices (such as webcams, video surveillance cameras), or other devices with digital imaging or video capabilities.

[0012] In general, the present disclosure describes image processing techniques for a digital camera having an image sensor and an image signal processor (ISP). The ISP may be configured to control the capture of image frames from one or more image sensors and process one or more image frames from the one or more image sensors to generate a view of a scene in a corrected image frame. The corrected image frame may be part of a sequence of image frames that forms a video sequence. The video sequence may include other image frames received from an image sensor or other image sensors and / or other corrected image frames based on input from an image sensor or another image sensor. In some embodiments, processing of one or more image frames may be performed within an image sensor, such as in a binning module. The image processing techniques described in the embodiments disclosed herein may be performed by circuits such as a binning module in an image sensor, in an image signal processor (ISP), in an application processor (AP), or in a combination of two or all of these components.

[0013] In one example, an image signal processor may receive instructions for capturing a sequence of image frames in response to loading of software (such as a camera application) to generate a preview display from an image capture device. The image signal processor may be configured to generate a single output frame stream based on image frames received from one or more image sensors. The single output frame stream may include raw image data from an image sensor, merged image data from an image sensor, or corrected image frames processed by one or more algorithms within the image signal processor (such as in a merging module). For example, image frames obtained from an image sensor (which may have some processing performed on the data before output to the image signal processor) may be processed in the image signal processor by processing the image frames through an image post-processing engine (IPE) and / or other image processing circuits for performing one or more of tone mapping, portrait lighting, contrast enhancement, gamma correction, etc.

[0014] After the image signal processor determines an output frame representing a scene using image correction (such as the merging described in various embodiments herein), the output frame may be displayed on a device display as a single still image and / or as part of a video sequence, saved to a storage device as a picture or video sequence, sent over a network, and / or printed to an output medium. For example, the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from different image sensors, and then generate corresponding output frames of image data (e.g., preview display frames, still image capture, frames for video, frames for object tracking, etc.). In other examples, the image signal processor may output frames of image data to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization (e.g., auto focus (AF), automatic white balance (AWB), and automatic exposure control (AEC)), generate video files via output frames, configure frames for display, configure frames for storage, send frames over a network connection, etc. That is, an image signal processor may obtain incoming frames from one or more image sensors, each of which is coupled to one or more camera lenses, and may then generate and output an output frame stream to various output destinations.

[0015] In some aspects, a corrected image frame may be produced by combining aspects of the image correction of the present disclosure with other computational photography techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR). In the case of HDR photography, a first image frame and a second image frame are captured using different exposure times, different apertures, different lenses, and / or other characteristics that may result in an improved dynamic range of the fused image when the two image frames are combined. In some aspects, the method may be performed for MFNR photography, wherein the first image frame and the second image frame are captured using the same or different exposure times, and the first image frame and the second image frame are fused to generate a corrected first image frame having reduced noise compared to the captured first image frame.

[0016] In some aspects, a device may include an image signal processor or processor (e.g., an application processor) that includes specific functionality for camera control and / or processing, such as enabling or disabling a merging module or otherwise controlling aspects of image correction. The methods and techniques described herein may be performed entirely by an image signal processor or processor, or various operations may be split between the image signal processor and the processor, and in some aspects split across additional processors.

[0017] The device may include one, two or more image sensors, such as including a first image sensor. When there are multiple image sensors, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a remote image sensor. In another example, the first sensor is configured to obtain an image through a first lens having a first optical axis, and the second sensor is configured to obtain an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. The configuration may occur in a lens cluster on a mobile device, such as where multiple image sensors and associated lenses are located in offset positions on the front or back side of the mobile device. Additional image sensors with larger, smaller, or the same field of view may be included. The image correction techniques described herein may be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0018] In an additional aspect of the present disclosure, a device configured for image processing and / or image capture is disclosed. The device includes a component for capturing an image frame. The device also includes one or more components for capturing data representing a scene, such as an image sensor (including a charge coupled device (CCD), a Bayer filter sensor, an infrared (IR) detector, an ultraviolet (UV) detector, a complementary metal oxide semiconductor (CMOS) sensor), a time-of-flight detector. The device may also include one or more components for gathering and / or focusing light into one or more image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses). These components can be controlled to capture a first image frame and / or a second image frame input to the image processing techniques described herein.

[0019] For those of ordinary skill in the art, other aspects, features, and specific implementations will become apparent when reviewing the following description of specific exemplary aspects in conjunction with the accompanying drawings. Although each feature may be discussed below with respect to certain aspects and accompanying drawings, each aspect may include one or more of the advantageous features discussed herein. In other words, although one or more aspects may be discussed as having certain advantageous features, one or more of such features may also be used according to each aspect. In a similar manner, although the exemplary aspects may be discussed below as aspects of a device, system, or method, the exemplary aspects may be implemented in various devices, systems, and methods.

[0020] The method may be embedded in a computer readable medium as a computer program code including instructions for causing a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device including: a first network adapter configured to send data, such as an image or video as recorded data or as streaming data, over a first network connection of a plurality of network connections; and a processor coupled to the first network adapter and a memory. The processor may cause the corrected image frame described herein to be sent over a wireless communication network, such as a 5G NR communication network.

[0021] The features and technical advantages of examples according to the present disclosure have been outlined quite extensively above so that the following specific embodiments may be better understood. Additional features and advantages will be described below. The disclosed concepts and specific examples may be easily used as a basis for modifying or designing other structures for achieving the same purpose of the present disclosure. Such equivalent constructions do not depart from the scope of protection of the appended claims. The characteristics of the concepts disclosed herein, both in terms of their organization and method of operation, and the associated advantages will be better understood by considering the following description in conjunction with the accompanying drawings. Each of the accompanying drawings is provided for the purpose of illustration and description, and not as a definition of limitations of the claims.

[0022] Although various aspects and specific implementations are described in this application by illustrating some examples, it will be understood by those skilled in the art that additional specific implementations and use cases may occur in many other arrangements and scenarios. The innovations described herein can be implemented across many different platform types, devices, systems, shapes, sizes, and packaging arrangements. For example, aspects and / or uses can be generated via integrated chip specific implementations and other devices based on non-module components (e.g., end-user devices, vehicles, communication devices, computing devices, industrial equipment, retail / purchase devices, medical devices, devices that enable artificial intelligence (AI), etc.). Although some examples may or may not specifically relate to use cases or applications, the innovations described may have various types of applicability. The scope of specific implementations may range from chip-level or module components to non-module, non-chip-level specific implementations, and further to aggregated, distributed, or original equipment manufacturer (OEM) devices or systems that combine one or more aspects of the described innovations. In some practical environments, the devices that combine the various aspects and features described may also necessarily include additional components and features for implementing and practicing the various aspects claimed and described. For example, the transmission and reception of wireless signals necessarily include multiple components for analog and digital purposes (e.g., hardware components including antennas, radio frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders / summers, etc.). The innovations described herein are intended to be practiced in a variety of devices, chip-level components, systems, distributed arrangements, end-user devices, etc., having different sizes, shapes, and configurations. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] A further understanding of the nature and advantages of the present disclosure may be achieved by reference to the following drawings. In the drawings, similar components or features may have the same reference numerals. In addition, various components of the same type may be distinguished by adding a dash and a second reference numeral after the reference numeral to distinguish between similar components. If only the first reference numeral is used in the specification, the description applies to any of the similar components having the same first reference numeral, regardless of the second reference numeral.

[0024] FIG. 1 shows a block diagram of an example device 100 for performing image capture from one or more image sensors.

[0025] Figure 2 is a block diagram illustrating an auto focus (AF) system using machine learning according to some embodiments of the present disclosure.

[0026] Figure 3 is a block diagram illustrating an autofocus system using machine learning with multiple available machine learning configurations according to some embodiments of the present disclosure.

[0027] Figure 4 is a block diagram illustrating a PDAF decision module for determining the application of machine learning in an autofocus system according to some embodiments of the present disclosure.

[0028] 5 is a flow chart illustrating a method of controlling the focus of a camera using machine learning according to some embodiments of the present disclosure.

[0029] Figure 6 is a flowchart illustrating a method of determining a machine learning configuration for determining a focal length of a camera according to some embodiments of the present disclosure.

[0030] FIG. 7 is an illustration of a method for determining weights for a machine learning algorithm according to some embodiments of the present disclosure.

[0031] 8 is a block diagram illustrating a feedback system for updating machine learning used in an autofocus system according to some embodiments of the present disclosure.

[0032] The same reference numbers and designations in different drawings represent the same elements. DETAILED DESCRIPTION

[0033] The specific embodiments described below in conjunction with the accompanying drawings are intended as descriptions of various configurations and are not intended to limit the scope of the present disclosure. Instead, the specific embodiments include specific details for providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in every case, and in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0034] The present disclosure provides systems, apparatuses, methods, and computer-readable media that support image capture operations. Autofocus (AF) performed by an image capture device may use a machine learning algorithm to determine information for assisting in controlling the image capture device to improve the focus of a photo or video captured by the image capture device. For example, the machine learning algorithm may use at least a portion of an image frame captured by the image capture device to determine a representation of the focal length of a scene (such as a phase shift value). A lens may be controlled based on the representation of the focal length to change the focus of the image capture device in order to improve the focus of additional captured image frames.

[0035] Specific implementations of the subject matter described in the present disclosure may be implemented to achieve one or more of the following potential advantages or benefits. In some aspects, the present disclosure provides techniques for improving autofocus (AF) by providing better focus information for obtaining image frames in better focus from an image sensor. In some aspects, the present disclosure provides techniques for improving autofocus (AF) by providing faster focus for obtaining image frames from an image sensor to reduce the delay experienced by a user of an image capture device. In low-light scenes, machine learning-based AF may provide more reliable and stable results than those obtainable without machine learning. In addition, reliable results may be obtained in challenging scene conditions such as moiré patterns and other repeating patterns and low light.

[0036] An example device for capturing image frames using one or more image sensors, such as a smartphone, may include a configuration of two, three, four, or more cameras on the back side (e.g., the side opposite the user display) or the front side (e.g., the same side as the user display) of the device. Devices with multiple image sensors include one or more image signal processors (ISPs), computer vision processors (CVPs) (e.g., an AI engine), or other suitable circuits for processing images captured by the image sensors. The one or more image signal processors may provide processed image frames to a memory and / or a processor (such as an application processor, an image front end (IFE), an image processing engine (IPE), or other suitable processing circuits) for further processing, such as for encoding, storage, transmission, or other manipulation.

[0037] As used herein, an image sensor may refer to the image sensor itself and any specific other components coupled to the image sensor for generating image frames for processing by an image signal processor or other logic circuitry or for storage in a memory (whether a short-term buffer or a long-term non-volatile memory). For example, an image sensor may include other components of a camera, including a shutter, a buffer, or other readout circuitry for accessing individual pixels of the image sensor. An image sensor may also refer to an analog front end or other circuitry for converting an analog signal into a digital representation of an image frame, which is provided to a digital circuit coupled to the image sensor.

[0038] In the following description, a large number of specific details (such as examples of specific components, circuits, and processes) are set forth in order to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected or connected through one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific terms are set forth in order to provide a thorough understanding of the present disclosure. However, it will be appreciated by those skilled in the art that these specific details may not be required to implement the teachings disclosed herein. In other examples, in order to avoid obscuring the teachings of the present disclosure, well-known circuits and devices are shown in block diagram form.

[0039] Some portions of the following detailed description are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, programs, logic blocks, processes, etc. are conceived as a self-consistent sequence of steps or instructions leading to a desired result. These steps are steps requiring physical manipulations of physical quantities. Typically, although not necessarily, these physical quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0040] In the accompanying drawings, a single box may be described as performing one or more functions. The one or more functions performed by the box may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, the following generally describes various exemplary components, blocks, modules, circuits, and steps in terms of their functionality. Whether such functionality is implemented as hardware or software depends on specific applications and the design constraints proposed for the entire system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing departure from the scope of the present disclosure. Moreover, the example device may include components other than those shown, including known components such as processors, memories, etc.

[0041] Aspects of the present disclosure are applicable to any electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or "frames"). In addition, aspects of the present disclosure may be implemented in image sensors or devices coupled to image sensors having the same or different capabilities and characteristics (e.g., resolution, shutter speed, sensor type, etc.). In addition, aspects of the present disclosure may be implemented in a device for processing image frames, whether or not the device includes or is coupled to an image sensor, such as a processing device that can retrieve stored images for processing, including a processing device present in a cloud computing system.

[0042] Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout this application, discussions utilizing terms such as "access," "receive," "transmit," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "set," "generate," etc., refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's registers, memories, or other such information storage, transmission, or display devices.

[0043] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (e.g., a smartphone, a camera controller, a processing system, etc.). As used herein, a device may be any electronic device having one or more parts that can implement at least some portions of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus may include a device or a portion of a device for performing the described operations.

[0044] FIG. 1 shows a block diagram of an example device 100 for performing image capture from one or more image sensors. The device 100 may include or otherwise be coupled to an image signal processor 112 for processing image frames from one or more image sensors, such as a first image sensor 101, a second image sensor 102, and a depth sensor 140. In some implementations, the device 100 also includes or is coupled to a processor 104 and a memory 106 storing instructions 108. The device 100 may also include or be coupled to a display 114 and an input / output (I / O) component 116. The I / O component 116 may be used to interact with a user, such as a touch screen interface and / or physical buttons. The I / O component 116 may also include a network interface for communicating with other devices, including a wide area network (WAN) adapter 152, a local area network (LAN) adapter 153, and / or a personal area network (PAN) adapter 154. An example WAN adapter is a 4G LTE or 5G NR wireless network adapter. An example LAN adapter 153 is an IEEE 802.11 WiFi wireless network adapter. An example PAN adapter 154 is a Bluetooth wireless network adapter. Each of the adapters 152, 153, and / or 154 may be coupled to an antenna including multiple antennas configured for primary set reception and diversity reception and / or configured for receiving a particular frequency band. The device 100 may also include or be coupled to a power source 118 for the device 100, such as a battery or a component that couples the device 100 to an energy source. The device 100 may also include or be coupled to additional features or components not shown in FIG. 1 . In one example, a wireless interface that may include multiple transceivers and a baseband processor may be coupled to the WAN adapter 152 or included in the WAN adapter for use in a wireless communication device. In yet another example, an analog front end (AFE) for converting analog image frame data into digital image frame data may be coupled between the image sensors 101 and 102 and the image signal processor 112.

[0045] The device may include or be coupled to a sensor hub 150 for interfacing with sensors to receive data about the movement of the device 100, data about the environment surrounding the device 100, and / or other non-camera sensor data. One example non-camera sensor is a gyroscope, i.e., a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, i.e., a device configured to measure acceleration, which may also be used to determine velocity and distance traveled by appropriately integrating the measured acceleration, and one or more of the acceleration, velocity, and / or distance may be included in the generated motion data. In some aspects, a gyroscope in an electronic image stabilization system (EIS) may be coupled to the sensor hub or directly to the image signal processor 112. In another example, the non-camera sensor may be a global positioning system (GPS) receiver.

[0046] The image signal processor 112 may receive image data such as for forming an image frame. In one embodiment, a local bus connection couples the image signal processor 112 to the image sensors 101 and 102 of the first camera and the second camera, respectively. In another embodiment, a wire interface couples the image signal processor 112 to an external image sensor. In yet another embodiment, a wireless interface couples the image signal processor 112 to the image sensors 101, 102.

[0047] The first camera may include a first image sensor 101 and a corresponding first lens 131. The second camera may include a second image sensor 102 and a corresponding second lens 132. Each of the lenses 131 and 132 may be controlled by an associated auto focus (AF) algorithm 133 executed in the ISP 112, which adjusts the lenses 131 and 132 to focus on a specific focal plane at a certain scene depth from the image sensors 101 and 102. The AF algorithm 133 may be assisted by a depth sensor 140.

[0048] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. The lenses 131 and 132 focus light at the image sensors 101 and 102, respectively, through one or more apertures for receiving light, one or more shutters for blocking light when outside the exposure window, one or more color filter arrays (CFAs) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurements into digital information, and / or other suitable components for imaging. The first lens 131 and the second lens 132 may have different fields of view to capture different representations of the scene. For example, the first lens 131 may be an ultra-wide (UW) lens, and the second lens 132 may be a wide (W) lens. The plurality of image sensors may include a combination of ultra-wide (high field of view (FOV)) sensors, wide sensors, long-range sensors, and ultra-long-range (low FOV) sensors. That is, each image sensor may be configured by hardware configuration and / or software settings to obtain different but overlapping fields of view. In one configuration, the image sensor is configured with different lenses having different magnifications, which results in different fields of view. The sensors may be configured such that the UW sensor has a larger FOV than the W sensor, the W sensor has a larger FOV than the T sensor, and the T sensor has a larger FOV than the UT sensor. For example, a sensor configured for a wide FOV may capture a field of view in the range of 64 to 84 degrees, a sensor configured for an ultra-lateral FOV may capture a field of view in the range of 100 to 140 degrees, a sensor configured for a long-range FOV may capture a field of view in the range of 10 to 30 degrees, and a sensor configured for an ultra-long-range FOV may capture a field of view in the range of 1 to 8 degrees.

[0049] Image signal processor 112 processes image frames captured by image sensors 101 and 102. Although FIG. 1 illustrates device 100 including two image sensors 101 and 102 coupled to image signal processor 112, any number (e.g., one, two, three, four, five, six, etc.) of image sensors may be coupled to image signal processor 112. In some aspects, a depth sensor such as depth sensor 140 may be coupled to image signal processor 112, and the output from the depth sensor may be processed in a manner similar to that of image sensors 101 and 102. Furthermore, any number of additional image sensors or image signal processors may be present for device 100.

[0050] In some embodiments, the image signal processor 112 may execute instructions from a memory, such as instructions 108 from the memory 106, instructions stored in a separate memory coupled to or included in the image signal processor 112, or instructions provided by the processor 104. In addition or in the alternative, the image signal processor 112 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described in the present disclosure. For example, the image signal processor 112 may include one or more image front ends (IFEs) 135, one or more image post-processing engines 136 (IPEs), and / or one or more automatic exposure compensation (AEC) 134 engines. The AF 133, AEC 134, IFE 135, IPE 136 may each include dedicated circuitry embodied as software code executed by the ISP 112 and / or a combination of hardware within the ISP 112 and software code executed on the ISP 112.

[0051] In some implementations, the memory 106 may include a non-transitory or non-transitory computer-readable medium storing computer-executable instructions 108 to perform all or part of one or more operations described in the present disclosure. In some implementations, the instructions 108 include a camera application (or other suitable application) to be executed by the device 100 for generating images or videos. The instructions 108 may also include other applications or programs executed by the device 100, such as an operating system and specific applications other than for image or video generation. A camera application such as that executed by the processor 104 may cause the device 100 to generate images using the image sensors 101 and 102 and the image signal processor 112. The memory 106 may also be accessed by the image signal processor 112 to store processed frames, or may be accessed by the processor 104 to obtain processed frames. In some embodiments, the device 100 does not include the memory 106. For example, the device 100 may be a circuit including the image signal processor 112, and the memory may be external to the device 100. The device 100 may be coupled to an external memory and configured to access the memory to write output frames for display or long-term storage. In some embodiments, device 100 is a system on a chip (SoC) that combines image signal processor 112 , processor 104 , sensor hub 150 , memory 106 , and input / output components 116 into a single package.

[0052] In some embodiments, at least one of the image signal processor 112 or the processor 104 executes instructions to perform various operations described herein, including noise reduction operations. For example, execution of the instructions may instruct the image signal processor 112 to start or end capturing an image frame or a sequence of image frames, wherein the capture includes noise reduction as described in the embodiments herein. In some embodiments, the processor 104 may include one or more general-purpose processor cores 104A capable of executing scripts or instructions of one or more software programs (such as instructions 108 stored in the memory 106). For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in the memory 106.

[0053] When executing the camera application, the processor 104 can be configured to command the image signal processor 112 to perform one or more operations with reference to the image sensor 101 or 102. For example, the camera application can receive a command to start a video preview display, and upon receiving the command, capture and process a video including a sequence of image frames from one or more image sensors 101 or 102. Image correction, such as using cascaded IPEs, can be applied to one or more image frames in the sequence. Execution of instructions 108 by the processor 104 outside the camera application can also cause the device 100 to perform any number of functions or operations. In some embodiments, in addition to the ability to execute software to cause the device 100 to perform multiple functions or operations (such as the operations described herein), the processor 104 can also include an IC or other hardware (e.g., an artificial intelligence (AI) engine 124). In some other embodiments, the device 100 does not include the processor 104, such as when all of the described functionality is configured in the image signal processor 112.

[0054] In some embodiments, display 114 may include one or more suitable displays or screens that allow user interaction and / or present items to the user (such as previews of image frames captured by image sensors 101 and 102). In some embodiments, display 114 is a touch-sensitive display. I / O component 116 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from a user and provide output to the user through display 114. For example, I / O component 116 may include (but is not limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (e.g., a power button), a slider, a switch, etc.

[0055] Although shown coupled to each other via processor 104, components such as processor 104, memory 106, image signal processor 112, display 114, and I / O component 116 may be coupled to each other in various other arrangements, such as via one or more local buses, which are not shown for simplicity. Although image signal processor 112 is illustrated as being separate from processor 104, image signal processor 112 may be a core of processor 104, which is an application processor unit (APU), included in a system on chip (SoC), or otherwise included in processor 104. Although device 100 is referenced in the examples herein to perform various aspects of the present disclosure, some device components may not be shown in FIG. 1 to prevent obscuring various aspects of the present disclosure. Additionally, other components, number of components, or combinations of components may be included in suitable devices for performing various aspects of the present disclosure. Therefore, the present disclosure is not limited to configurations of specific devices or components, including device 100.

[0056] The AF algorithm 133 of the image capture device 100 may be based on the reference Figure 2 to FIG. 7 for some or all aspects of the embodiment described. Figure 21 is a block diagram illustrating an autofocus (AF) system using machine learning according to some embodiments of the present disclosure. The first image sensor 101 of the camera 103 outputs image data 202, which may include phase shift information and / or a representation of a scene in the field of view of the image sensor 101. The phase shift information may include sensor data from one or more phase shift sensors embedded in the first image sensor 101. The representation of the scene may include Bayer formatted image data, although other image formats may be used to represent the scene. The ISP 112 is coupled to the first camera 103 to receive the image data 202, such as by coupling the image sensor 101 to a bus of the ISP 112. Although the functionality may be described as part of the ISP 112, other components may be configured to perform similar functionality in controlling the image capture device and providing autofocus capabilities in the first camera 103. For example, a processor (such as the processor 104) or other general or fixed function logic circuits may be configured to provide the autofocus functionality described herein. In some embodiments, the autofocus functionality may be performed by the ISP 112 in conjunction with other processing circuits (such as the AI ​​engine 124). For example, the AI ​​engine 124 may be embedded in the processor 104 and coupled to the ISP 112 via a bus. As another example, the AI ​​engine 124 may be a separate component in the device 100 and coupled to the ISP 112 via a bus. As another example, the AI ​​engine 124 may be embedded in the ISP 112. In some embodiments, one or more of the ISP 112, the processor 104, and the AI ​​engine 124 may be co-located as a single integrated circuit (IC) on a shared substrate that may be shared with other functionality including one or more of the sensor hub 150, the memory 106, and / or the I / O components 116.

[0057] ISP 112 outputs the focus information to first camera 103 for controlling the focus of first camera 103. Camera 103 may use the focus information to adjust the position of first lens 131. ISP 112 may be coupled via the same bus that transmits image data 202, via a different bus, or via other control signals. In some embodiments, the focus information may include a focal length determined by ISP 112 as an estimated focus of a scene in the field of view of camera 103. Camera 103 may control lens 131 according to the received focal length by directly setting the lens position to the focal length. In some embodiments, the focus information may include different data used by camera 103 to determine the position of lens 131 and adjust the position of lens 131 according to the determined position. For example, the focus information may be a focus shift value indicating a distance and / or direction for moving the position of lens 131. Camera 103 may determine a new absolute position value of lens 131 based on the current position value and the focus shift value, and adjust lens 131 based on the new absolute position value.

[0058] The autofocus (AF) algorithm 133 executed by the ISP 112 may include several components, including a traditional phase detection autofocus (PDAF) module 212A, a PDAF decision module 212B, and a machine learning phase detection (MLPD) module 212C. The traditional PDAF module 212A may include one or more non-machine learning AF algorithms, such as a phase detection AF algorithm that estimates the amount by which one image arriving at the PD sensor is shifted from another image arriving at the PD sensor. In some embodiments, the traditional PDAF module 212A may include a cross-correlation determination between images. The PDAF decision module 212B may determine how to determine the focus information sent to the camera 103. For example, the PDAF decision module 212B may determine whether to use one or both of the traditional PDAF module 212A and the MLPD module 212C to determine the focus information. As another example, the PDAF decision module 212B may determine how to use the MLPD module 212C to adjust the focus information generated by the traditional PDAF module 212A and / or how to use the traditional PDAF module 212A to adjust the focus information generated by the MLPD module 212C. In some embodiments, the AF algorithm 133 may include one or some of the modules 212A, 212B, 212C, and may include other modules for performing autofocus functions. For example, the AF algorithm 133 may perform autofocus using only machine learning in the MLPD module 212C for determining the focus information 204, so that the AF algorithm 133 does not include modules 212A and 212B. As another example, the AF algorithm 133 may include a tracking algorithm, such as object tracking or (specifically) face tracking. Object tracking may cooperate with one or both of the conventional PDAF module 212A and the MLPD module 212C for determining the focus information.

[0059] Figure 3 One configuration of the AF algorithm 133 having modules 212A, 212B, and 212C is shown in FIG. Figure 3is a block diagram illustrating an autofocus system using machine learning with multiple available machine learning configurations according to some embodiments of the present disclosure. The AF algorithm 133 may receive phase shift information corresponding to a left image and a right image corresponding to a first scene from a phase detector (PD) buffer, and may additionally receive raw image data corresponding to a representation of the first scene. The conventional PDAF module 212A may determine first focus information, which is provided to the PDAF decision module 212B. The PDAF decision module 212B may determine whether to provide the first focus information as an output of the AF algorithm based on one or more criteria. For example, one criterion may specify that the AF algorithm 133 outputs the first focus information when a confidence level associated with the first focus information exceeds a threshold confidence level. As another example, a separate or additional criterion may specify that the AF algorithm 133 outputs the first focus information when a specific scene condition is detected (e.g., when an object is detected, when a specific object is detected, or when a specific light level is detected).

[0060] The MLPD module 212C may determine the second focus information to replace or adjust the first focus information. The PDAF decision module 212B may determine whether to provide the second focus information as an output of the AF algorithm based on one or more criteria. For example, one criterion may specify that the output of the AF algorithm 133 is the second focus information when a specific scene condition is detected (such as when a light level below a threshold level is detected or when the confidence level of the PDAF module 212A is below a threshold level). In some embodiments, the MLPD module 212C uses filters from a convolutional layer to determine the second focus information for noise reduction, which enables better focus information, particularly in noisy images captured in low-light scenes, although the second focus information may be available in all scenes.

[0061] The MLPD module 212C may be activated and deactivated by the PDAF decision module 212B to reduce power consumption during execution of the AF algorithm 133. The MLPD module 212C may consume more power than the conventional PDAF module 212A, and thus deactivating the MLPD module 212C may reduce power consumption when the conventional PDAF module 212A provides sufficient focusing capabilities. The reduced power consumption may be particularly advantageous when the image capture device executing the AF algorithm 133 is operated on battery power (such as by a mobile device). Additionally, deactivating the MLPD module 212C may reduce the processing resources consumed, thereby making the processing resources available for use by other algorithms to the extent that the processing resources are shared between the AF algorithm 133 and the other algorithms. Furthermore, the MLPD module 212C may add a delay to the determination of the focusing information, which may be eliminated when the scene conditions allow the conventional PDAF module 212A to provide sufficient focusing information.

[0062] The MLPD module 212C may include an MLPD preprocessing module 312 and an MLPD decision module 314. The MLPD preprocessing module 312 may process image data representing a scene to obtain input to a machine learning algorithm. For example, a machine learning algorithm may be configured to receive a fixed size image as an input for prediction of focus information. In some embodiments, the machine learning algorithm is a neural network trained with a specific image size, and therefore the image data may be preprocessed to obtain samples of the same image size used to train the neural network. Preprocessing may include cropping, resizing, padding, or other image processing of the image data to determine the input to the machine learning algorithm. In some embodiments, preprocessing may include determining a region of interest (ROI) in a scene corresponding to a trained size and cropping the image data to the ROI for input to the machine learning algorithm. For example, the ROI may be determined by object recognition and / or user input. For example, an image capture device may allow a user to specify a region of interest by touching a location and / or defining an area corresponding to a desired focus on a display. As another example, pre-processing may include determining a ROI in a scene, cropping the image data to the ROI, and then resizing and / or padding the cropped image data to match a trained size for input to a machine learning algorithm.

[0063] In some embodiments, a single ROI may be applied to the MLPD module 212C to output multiple focus results. For example, a region of interest corresponding to a window surrounding a portion of the image data may be divided into a plurality of N sub-windows, such as a 4×3 array of sub-windows. A weighted average may be determined from the N sub-windows and used to determine focus information for the ROI based on the N sets of statistics determined from the N sub-windows.

[0064] The MLPD decision module 314 may determine the configuration of the machine learning algorithm when determining the focus information. This may improve the focus results in certain scene conditions that are difficult for phase detection algorithms. For example, the focus on a moire pattern or a repeating pattern in a scene may be improved using a machine learning algorithm that is specifically trained for image scenes that include such patterns.

[0065] Figure 4 An example implementation of a standard PDAF decision module 212B applying the operation of an AF algorithm is shown in FIG. Figure 44 is a block diagram illustrating a PDAF decision module for determining the application of machine learning in an autofocus system according to some embodiments of the present disclosure. At box 402, the PDAF decision module 212B receives traditional PDAF focus information and determines whether the focus information has a confidence level above a threshold level. If so, at box 406, the traditional PDAF focal length is sent to the camera as a first focal length. For example, at box 406, the ISP 112 executing the PDAF decision module 212B may send a command to the camera to focus at a focal length corresponding to the output of the traditional PDAF module 212A. If the confidence level is below the confidence threshold, at box 404, the image data is provided to the MLPD module 212C and a second focal length is determined based on machine learning.

[0066] A method for performing autofocus on an image capture device according to an embodiment described herein is shown in FIG. 5. FIG. 5 is a flowchart illustrating a method for controlling the focal length of a camera using machine learning according to some embodiments of the present disclosure. Method 500 includes receiving first image data including phase shift information at box 502. The first image data may be received from a first image sensor of a first camera of an image capture device. The first image sensor may include a plurality of phase detectors at different positions to record corresponding phase information pairs of left and right images. The phase detector may be embedded in an image sensor configured to obtain image data including a representation of a scene. At box 504, a first focal length for a first scene is determined based on a machine learning algorithm. For example, the MLPD module 212C may be used to determine the first focal length as focus information. The determination at box 504 may include, for example, determining by the PDAF decision module 212B whether to use the MLPD module 212C. At box 506, the focal length of the first camera is controlled based on the first focal length determined at box 504 using machine learning.

[0067] Determining a first focal length for a first scene at block 504 may include: Figure 6 Configure machine learning based on scenario conditions as shown in . Figure 6FIG. 0 is a flow chart illustrating a method of determining a machine learning configuration for determining a focal length of a camera in accordance with some embodiments of the present disclosure. Method 600 may include determining at block 602 whether the scene includes a repeating pattern. If so, then at block 604 the machine learning is configured by loading weights trained from the repeating pattern. If not, then at block 606, method 600 includes determining whether the scene condition reflects normal usage. If so, then at block 608 the machine learning is configured by loading weights trained from normal usage scenes. If not, then at block 610, method 600 includes determining whether the scene condition reflects a multi-depth scene. If so, then at block 612 the machine learning is configured by loading weights trained from multi-depth low light scenes. In some embodiments, the weights of block 612 may be determined by a weighted average, such as described with reference to FIG. 7. If not, then at block 614, method 600 includes configuring the machine learning by loading weights trained from single-depth low light scenes. After loading the weights according to one of blocks 604, 608, 612, and 614, method 600 includes determining a focus shift and a confidence level using the machine learning in MLPD module 212C at block 616.

[0068] FIG. 7 is an illustration of determining weights for a machine learning algorithm in accordance with some embodiments of the present disclosure. Image portion 702 may extend from coordinates (x0, y0) in one corner of exemplary image portion 702 to coordinates (x4, y3) in the opposite corner, the exemplary image portion being divided into 12 sub-windows arranged in a 4×3 grid, each sub-window having its own determined focal length. Image portion 702 may represent a fixed-size image that the machine learning algorithm is configured to receive by training with training images of the same fixed-size image as image portion 702. Region of interest (ROI) 704 for focusing may be a smaller portion of image portion 702, such as a face portion of a larger image portion. ROI 704 may be a floating window that moves around within fixed-size image portion 702 corresponding to movement of an object of interest within image portion 702. ROI 704 may be defined by coordinates (fx0, fy0) in one corner and coordinates (fx1, fy1) in the opposite corner, where x0 < fx0 < x4 and y0 < fy0 < y3 and x0 < fx1 < x4 and y0 < fy1 < y3.

[0069] The phase difference (PD) of ROI 704 may be determined by a weighted average of the depth of focus within each of the sub-windows within image portion 702 based on the amount of overlap of ROI 704 with each of the sub-windows. As an example of one portion 706, the overlap may be determined based on the following equation:

[0070]

[0071] The weights for other portions of the ROI 704 may be similarly determined. The weights may be applied to use the weight weight of each sub-window of the image portion 702 based on the following equation: i,j The output MLPD of the MLPD module 212C for each sub-window of the image portion 702 is i,j To determine the phase difference of ROI 704:

[0072]

[0073] The weighting allows the overlapping area of ​​ROI 704 and image portion 702 to have a larger influence on the determined phase difference and allows the non-overlapping area to have zero weight so that the final MLPD output can be dominated by ROI 704 .

[0074] The machine learning algorithm used to determine the focus information may be trained to determine the set of weights through offline training. For example, a sample data set including images and associated ground truth may be used to determine weights that may be used for configuration at blocks 604, 608, 612, and 614. As another example, the weights may be retrieved from a configuration file, and the configuration file may be retrieved from a remote server.

[0075] Feedback may be used during operation of the image capture device to update a machine learning algorithm for autofocus, as shown in FIG8 . FIG8 is a block diagram illustrating a feedback system for updating machine learning used in an autofocus system according to some embodiments of the present disclosure. Method 800 includes operating the MLPD module 212C to generate a prediction 804. The prediction 804 may be determined during the capture of image data by the image capture device, so that the machine learning algorithm is updated in real time during use of the image capture device. Alternatively, aspects of method 800 may be performed offline, such as when the image capture device is plugged in. Phase detection autofocus (PDAF) is performed at box 802 based on a grid or other condition to provide a ground truth. At box 806, the ground truth of box 802 and the prediction of box 804 are compared, such as by calculating a loss for focus shift and an associated confidence level, wherein the calculated loss indicates the difference between the ground truth and the prediction. The calculation of box 806 is used to update the stored weights at box 808, wherein the MLPD module 212C uses the updated weights to determine future focus information.

[0076] In one or more aspects, techniques for supporting image processing (such as in supporting image capture devices) may include additional aspects, such as any single aspect or any combination of aspects described below or in combination with one or more other processes or devices described elsewhere herein. In a first aspect, supporting image processing may include an apparatus configured to perform operations including: receiving first image data of a first scene from a first image sensor of a first camera, the first image data including phase shift information; determining a first focal length for the first scene based on a machine learning algorithm by inputting the phase shift information into the machine learning algorithm; and controlling a focus position of the first camera based on the first focal length. Additionally, the apparatus may perform or operate according to one or more aspects as described below.

[0077] In some specific implementations, the apparatus includes a wireless device, such as a UE, configured with one or more cameras and configured to communicate via a wireless network. In some specific implementations, the apparatus may include at least one processor and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the apparatus. In some other specific implementations, the apparatus may include a non-transitory computer-readable medium having a program code recorded thereon, and the program code may be executable by a computer to cause the computer to perform the operations described herein with reference to the apparatus. In some specific implementations, the apparatus may include one or more components configured to perform the operations described herein. In some specific implementations, the method of wireless communication may include one or more operations described herein with reference to the apparatus.

[0078] In a second aspect, in combination with the first aspect, the first image data also includes a representation of the first scene, and the device is configured to perform operations including: determining characteristics of the first scene based on the first image data; and configuring the machine learning algorithm based on the characteristics of the first scene.

[0079] In a third aspect, in combination with one or more of the first aspect or the second aspect, configuring the machine learning algorithm includes loading multiple weights based on the characteristics of the first scene, wherein the characteristics of the first scene include at least one of a repeating pattern scene, a multi-depth scene, or a low-light scene.

[0080] In a fourth aspect, in combination with one or more of the first to third aspects, the device is further configured to perform operations including: determining a second focal length based on the phase shift information without the machine learning algorithm; and determining a confidence level associated with the second focal length.

[0081] In a fifth aspect, in combination with one or more of the first to fourth aspects, the focus position of the first camera is controlled based on the confidence level, so that when a first criterion of the confidence level is met, the focus position of the first camera is controlled based on the first focal length, and when a second criterion of the confidence level is met, the focus position of the first camera is controlled based on the second focal length.

[0082] In a sixth aspect, in combination with one or more of the first to fifth aspects, determining the first focal length is based on satisfying the first criterion of the confidence level, such that when the first criterion is not satisfied, the first focal length is not determined.

[0083] In a seventh aspect, in combination with one or more of aspects 1 to 6, receiving the first image data also includes receiving a representation of the first scene, wherein determining the first focal length based on the machine learning algorithm also includes inputting at least a portion of the representation of the first scene into the machine learning algorithm.

[0084] In an eighth aspect, in combination with one or more of aspects one to seven, the device is further configured to perform operations including: processing the first image data to determine a window of the first image data based on a region of interest, wherein determining the first focal length based on the machine learning algorithm includes inputting the window of the first image data into the machine learning algorithm, wherein a size of the window corresponds to an input size of the machine learning algorithm.

[0085] In a ninth aspect, in combination with one or more of the first to eighth aspects, the apparatus is further configured to perform operations comprising: determining a plurality of sub-windows of the area of ​​interest, wherein determining the first focal length for the first scene comprises determining a plurality of first focal lengths based on the machine learning algorithm by inputting the plurality of sub-windows, and wherein controlling the focus position of the first camera is based on the plurality of first focal lengths.

[0086] In a tenth aspect, in combination with one or more of the first to ninth aspects, controlling the focus position of the first camera is based on a weighted average of the plurality of first focal lengths.

[0087] In the eleventh aspect, in combination with one or more of the first to tenth aspects, the device is further configured to perform operations including: determining a second focal length based on the phase shift information without the machine learning algorithm; and updating the machine learning algorithm based on the first focal length and the second focal length.

[0088] In a twelfth aspect, in combination with one or more of the first to eleventh aspects, the device includes a first camera having the first image sensor and a lens, wherein the device is configured with an autofocus (AF) algorithm to control the position of the lens based on a determined focal length (such as the focal length determined by the machine learning algorithm).

[0089] It should be understood by those skilled in the art that any of a variety of different technologies and techniques may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0090] The components, functional blocks and modules described herein with respect to Figures 1 to 8 include processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software codes, firmware codes, etc., or any combination thereof. Software should be broadly interpreted as meaning instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, and / or functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language or other terms. In addition, the features discussed herein can be implemented via dedicated processor circuits, via executable instructions or their combinations.

[0091] The technician will further understand that the various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the present disclosure can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, frames, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. Those skilled in the art can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing departure from the scope of the present disclosure. The technician will also easily recognize that the order or combination of the components, methods or interactions described herein are merely examples, and the components, methods or interactions of various aspects of the present disclosure can be combined or performed in a manner other than the manner illustrated and described herein.

[0092] The various illustrative logical components, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. The interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0093] The hardware and data processing apparatus for implementing the various illustrative logic components, logic boxes, modules and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller or state machine. In some specific implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. In some specific implementations, specific processes and methods may be performed by circuits specific to a given function.

[0094] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, including the structures disclosed in this specification and their structural equivalents, or in any combination thereof. Specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on computer storage media for execution by data processing apparatus or for controlling the operation of data processing apparatus.

[0095] If implemented in software, the function may be stored as one or more instructions or codes on a computer-readable medium or sent through a computer-readable medium. The process of the method or algorithm disclosed herein may be implemented in a processor-executable software module that may reside on a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that can be implemented to transfer a computer program from one place to another. The storage medium may be any available medium that a computer can access. As an example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the required program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection may be appropriately referred to as a computer-readable medium. Disks and optical disks as used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and blue-ray disks, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media. Additionally, the operations of a method or algorithm may be located as a code and instruction set, or any combination of code and instruction sets, on a machine-readable medium or computer-readable medium, which may be incorporated into a computer program product.

[0096] Various modifications to the specific implementations described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to some other specific implementations without departing from the spirit or scope of the disclosure. Therefore, the claims are not intended to be limited to the specific implementations shown herein, but are to be consistent with the broadest scope consistent with the disclosure, the principles and novel features disclosed herein.

[0097] Additionally, one of ordinary skill in the art will readily appreciate that the terms "upper" and "lower" are sometimes used for ease of describing the drawings and indicate relative positions on a correctly oriented page corresponding to the orientation of the drawings, and may not reflect the correct orientation of any device as implemented.

[0098] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be removed from the combination, and a claimed combination may be directed to a subcombination or variations of a subcombination.

[0099] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all illustrated operations to achieve the desired result. In addition, the accompanying drawings can schematically depict one or more example processes in the form of a flow chart. However, other operations that are not depicted can be incorporated into the schematically illustrated example process. For example, one or more additional operations can be performed before, after, simultaneously or between any illustrated operations. In some cases, multitasking and parallel processing are advantageous. In addition, the separation of various system components in the specific implementation described above should not be understood as requiring such separation in all specific implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other specific implementations also fall within the scope of the appended claims. In some cases, the actions narrated in the claims can be performed in different orders and still achieve the desired result.

[0100] As used herein, including in the claims, the term "or" used in a list of two or more items means that any of the listed items may be taken alone, or any combination of two or more of the listed items may be taken. For example, if a composition is described as comprising components A, B, or C, the composition may comprise A alone; B alone; C alone; A and B in combination; A and C in combination; B and C in combination; or A, B, and C in combination. In addition, as used herein, including in the claims, "or" as used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination of any of these. The term "substantially" is defined as to a large extent, but not necessarily entirely, what is specified (and includes what is specified; for example, substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by one of ordinary skill in the art. In any disclosed implementation, the term "substantially" may be replaced with "within [percent] of" a specified content, where the percentage includes 0.1%, 1%, 5%, or 10%.

[0101] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Therefore, the disclosure is not intended to be limited to the examples and designs described herein, but to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method, comprising: receiving first image data of a first scene from a first image sensor of a first camera, the first image data comprising phase shift information; determining a first focal length for the first scene based on a machine learning algorithm by inputting the phase shift information into the machine learning algorithm; as well as A focus position of the first camera is controlled based on the first focal length.

2. The method of claim 1 , wherein the first image data further comprises a representation of the first scene, the method further comprising: determining a characteristic of the first scene based on the first image data; as well as The machine learning algorithm is configured based on the characteristics of the first scenario.

3. The method of claim 2, wherein configuring the machine learning algorithm comprises loading a plurality of weights based on the characteristics of the first scene, wherein the characteristics of the first scene comprise at least one of a repeating pattern scene, a multi-depth scene, or a low-light scene.

4. The method according to claim 1, further comprising: determining a second focal length based on the phase shift information without the machine learning algorithm; as well as determining a confidence level associated with the second focal length, Wherein controlling the focus position of the first camera is based on the confidence level, such that when a first criterion of the confidence level is met, controlling the focus position of the first camera is based on the first focal length, and when a second criterion of the confidence level is met, controlling the focus position of the first camera is based on the second focal length. The method of claim 4 , wherein determining the first focal length is based on satisfying the first criterion of the confidence level, such that when the first criterion is not satisfied, the first focal length is not determined.

6. The method of claim 1 , wherein receiving the first image data further comprises receiving a representation of the first scene, wherein determining the first focal length based on the machine learning algorithm further comprises inputting at least a portion of the representation of the first scene into the machine learning algorithm.

7. The method according to claim 6, further comprising: processing the first image data to determine a window of the first image data based on a region of interest, wherein determining the first focal length based on the machine learning algorithm comprises inputting the window of the first image data to the machine learning algorithm, The size of the window corresponds to the input size of the machine learning algorithm.

8. The method according to claim 7, further comprising: determining a plurality of sub-windows of the region of interest, wherein determining the first focal length for the first scene comprises determining a plurality of first focal lengths based on the machine learning algorithm by inputting the plurality of sub-windows into the machine learning algorithm, and Wherein controlling the focus position of the first camera is based on the plurality of first focal lengths. 9 . The method of claim 8 , wherein controlling the focus position of the first camera is based on a weighted average of the plurality of first focal lengths.

10. The method according to claim 1, further comprising: determining a second focal length based on the phase shift information without the machine learning algorithm; as well as The machine learning algorithm is updated based on the first focal length and the second focal length.

11. A device, comprising: a memory storing processor readable code; and at least one processor, the at least one processor being coupled to the memory, the at least one processor being configured to execute the processor readable code so that the at least one processor performs operations including: receiving first image data of a first scene from a first image sensor of a first camera, the first image data comprising phase shift information; determining a first focal length for the first scene based on a machine learning algorithm by inputting the phase shift information into the machine learning algorithm; as well as A focus position of the first camera is controlled based on the first focal length.

12. The apparatus of claim 11, wherein the first image data further comprises a representation of the first scene, and wherein the at least one processor is configured to execute the processor readable code to cause the at least one processor to perform further operations comprising: determining a characteristic of the first scene based on the first image data; and The machine learning algorithm is configured based on the characteristics of the first scenario.

13. The apparatus of claim 12, wherein configuring the machine learning algorithm comprises loading a plurality of weights based on the characteristics of the first scene, wherein the characteristics of the first scene comprise at least one of a repeating pattern scene, a multi-depth scene, or a low-light scene.

14. The apparatus of claim 11, wherein the at least one processor is configured to execute the processor readable code to cause the at least one processor to perform further operations comprising: determining a second focal length based on the phase shift information without the machine learning algorithm; and determining a confidence level associated with the second focal length, Wherein controlling the focus position of the first camera is based on the confidence level, such that when a first criterion of the confidence level is met, controlling the focus position of the first camera is based on the first focal length, and when a second criterion of the confidence level is met, controlling the focus position of the first camera is based on the second focal length.

15. The apparatus of claim 14, wherein determining the first focal length is based on satisfying the first criterion of the confidence level, such that when the first criterion is not satisfied, the first focal length is not determined.

16. The apparatus of claim 11, wherein receiving the first image data further comprises receiving a representation of the first scene, wherein determining the first focal length based on the machine learning algorithm further comprises inputting at least a portion of the representation of the first scene into the machine learning algorithm.

17. The apparatus of claim 16, wherein the at least one processor is configured to execute the processor readable code to cause the at least one processor to perform further operations comprising: processing the first image data to determine a window of the first image data based on a region of interest, wherein determining the first focal length based on the machine learning algorithm comprises inputting the window of the first image data to the machine learning algorithm, The size of the window corresponds to the input size of the machine learning algorithm.

18. The apparatus of claim 17, wherein the at least one processor is configured to execute the processor readable code to cause the at least one processor to perform further operations comprising: determining a plurality of sub-windows of the region of interest, wherein determining the first focal length for the first scene comprises determining a plurality of first focal lengths by inputting the plurality of sub-windows based on the machine learning algorithm, and Wherein controlling the focus position of the first camera is based on the plurality of first focal lengths.

19. The apparatus of claim 18, wherein controlling the focus position of the first camera is based on a weighted average of the plurality of first focal lengths.

20. The apparatus of claim 11, wherein the at least one processor is configured to execute the processor readable code to cause the at least one processor to perform further operations comprising: determining a second focal length based on the phase shift information without the machine learning algorithm; and The machine learning algorithm is updated based on the first focal length and the second focal length.

21. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving first image data of a first scene from a first image sensor of a first camera, the first image data comprising phase shift information; determining a first focal length for the first scene based on a machine learning algorithm by inputting the phase shift information into the machine learning algorithm; and A focus position of the first camera is controlled based on the first focal length.

22. The non-transitory computer readable medium of claim 21, wherein the first image data further comprises a representation of the first scene, and wherein the non-transitory computer readable medium causes the processor to perform further operations comprising: determining a characteristic of the first scene based on the first image data; and The machine learning algorithm is configured based on the characteristics of the first scenario.

23. The non-transitory computer-readable medium of claim 22, wherein configuring the machine learning algorithm comprises loading a plurality of weights based on the characteristics of the first scene, wherein the characteristics of the first scene comprise at least one of a repeating pattern scene, a multi-depth scene, or a low-light scene.

24. The non-transitory computer readable medium of claim 21, wherein the operations further comprise: determining a second focal length based on the phase shift information without the machine learning algorithm; as well as determining a confidence level associated with the second focal length, Wherein controlling the focus position of the first camera is based on the confidence level, such that when a first criterion of the confidence level is met, controlling the focus position of the first camera is based on the first focal length, and when a second criterion of the confidence level is met, controlling the focus position of the first camera is based on the second focal length.

25. The non-transitory computer readable medium of claim 24, wherein determining the first focal length is based on satisfying the first criterion of the confidence level, such that when the first criterion is not satisfied, the first focal length is not determined.

26. An apparatus, comprising: a first camera, the first camera comprising a first image sensor; a memory storing processor readable code; and at least one processor coupled to the memory and to the first camera, the at least one processor configured to execute the processor readable code so that the at least one processor: receiving first image data of a first scene from the first image sensor, the first image data comprising phase shift information; determining a phase detection autofocus (PDAF) focal length based on the first image data and a confidence level associated with the PDAF focal length; When the confidence level is above a threshold level, controlling a focus position of the first camera based on the PDAF focal length; as well as When the confidence level is not higher than the threshold level: determining a machine learning phase detection (MLPD) focal length based on a machine learning algorithm by inputting at least a portion of the first image data into the machine learning algorithm; as well as The focus position of the first camera is controlled based on the MLPD focal length.

27. The apparatus of claim 26, wherein the first image data further comprises a representation of the first scene, and wherein the at least one processor is configured to execute the processor readable code to further cause the at least one processor to perform operations comprising: determining a characteristic of the first scene based on the first image data; and The machine learning algorithm is configured based on the characteristics of the first scenario.

28. The apparatus of claim 26, wherein the at least one processor is configured to execute the processor readable code to further cause the at least one processor to perform operations comprising: processing the first image data to determine a window of the first image data based on a region of interest, wherein determining the MLPD focal length comprises inputting the window of the first image data to the machine learning algorithm, The size of the window corresponds to the input size of the machine learning algorithm.

29. The apparatus of claim 28, wherein the at least one processor is configured to execute the processor readable code to further cause the at least one processor to perform operations comprising: determining a plurality of sub-windows of the region of interest, wherein determining the MLPD focal length comprises determining a plurality of first focal lengths based on the machine learning algorithm by inputting the plurality of sub-windows into the machine learning algorithm respectively, and Wherein controlling the focus position of the first camera is based on the plurality of first focal lengths.

30. The apparatus of claim 29, wherein controlling the focus position of the first camera is based on a weighted average of the plurality of first focal lengths.

Images