On-demand phase detection autofocus (PDAF)

Abstract

In one embodiment, a system includes: one or more memories configured to store camera images captured with a camera; and processing circuitry in communication with the one or more memories. The processing circuitry is configured to identify a first camera image portion and a second camera image portion in the camera image, apply primary phase detection autofocus (PDAF) to the first camera image portion, and apply primary PDAF and secondary PDAF to the second camera image portion. Additionally, the processing circuitry is configured to control the camera to focus the first camera image portion and the second camera image portion based on applying the primary PDAF to the first camera image portion and not applying the secondary PDAF to the first camera image portion and based on applying the primary PDAF and the secondary PDAF to the second camera image portion.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 462,230, filed September 6, 2023, the entirety of which is hereby incorporated by reference herein. TECHNICAL FIELD

[0002] The present disclosure relates to capturing or image processing. BACKGROUND

[0003] A camera device includes one or more cameras that capture frames (e.g., images). Examples of camera devices include a standalone digital camera or digital video camcorder, a wireless communication device handset equipped with a camera such as a mobile phone having one or more cameras, a cellular or satellite radio telephone, a personal digital assistant (PDA) equipped with a camera, a computing panel or tablet, a gaming device, a computer device including a camera such as a so-called “webcam,” a smart watch, a device equipped with a camera, a device configured to control another device equipped with a camera, or any device with digital imaging or video capabilities. The camera device processes the captured frames and outputs the frames for display. In some examples, the camera device controls exposure, focus, and white balance to capture high quality images.

[0004] In some examples, a camera can use phase detection auto focus (PDAF) techniques to automatically focus on one or more objects. PDAF involves processing camera data to identify whether an image is out of focus. When an image is out of focus, the camera can use PDAF to automatically control the focus of the camera. For example, the camera can adjust the distance between one or more camera lenses and a camera sensor to adjust the focus of the camera. SUMMARY

[0005] Generally, the present disclosure describes techniques for performing phase detection auto focus (PDAF) in a manner that conserves computational resources by using specialized image processing on portions (e.g., only portions) of a camera image that main PDAF image processing techniques cannot adequately process using. The system can apply main PDAF image processing to a camera image. But in some cases, the main PDAF image processing is inadequate to process one or more portions of the camera image. When the main PDAF image processing is inadequate, the system can apply secondary PDAF image processing to adequately process one or more portions of the camera image that the main PDAF could not adequately process. This can ensure that the system only uses computational resources beyond those used to perform main PDAF image processing when the main PDAF image processing produces inadequate results.

[0006] In some examples, the primary PDAF image processing can include using a hardware level image processing unit to perform the level PDAF image processing. For example, the system can use the hardware level image processing unit to apply the level PDAF image processing to the camera image. In some examples, the level PDAF image processing can not be sufficient to process some regions of the camera image. The system can apply the vertical PDAF image processing to the regions of the camera image that were not sufficiently focused using the level image processing. In some examples, the system can use a hardware vertical image processing unit to perform the vertical PDAF image processing. In some examples, the system can use software and / or one or more machine learning models to perform the vertical PDAF image processing.

[0007] Some camera images can be generated using multiple exposures. For example, a camera image can include a long exposure component and a short exposure component. In some examples, the primary PDAF image processing can include using a hardware unit to perform level PDAF image processing on the long exposure component of the camera image. But in some cases, using a hardware unit to perform level PDAF image processing on the long exposure component of the camera image can not be sufficient to process the camera image. The system can use software executing on processing circuitry to process the short exposure component based on the level PDAF image processing on the long exposure component using the hardware unit being insufficient.

[0008] Compared to other systems that perform PDAF image processing, the techniques of the present disclosure can result in more efficient use of computational resources for PDAF image processing. For example, by applying primary PDAF image processing and using secondary PDAF image processing only when needed to achieve sufficient results, the system can reduce the amount of computational resources needed for camera focus compared to systems that do not process data based on whether secondary resources are needed. Reducing the amount of computational resources needed for camera focus can reduce the amount of time required to process a camera image. Additionally or alternatively, reducing the amount of computational resources needed for camera focus can reduce the amount of energy consumed by the circuitry to process the camera image.

[0009] In one example, a system includes one or more memories configured to store a camera image captured with a camera and processing circuitry in communication with the one or more memories. The processing circuitry is configured to identify a first camera image portion and a second camera image portion in the camera image, apply PDAF to the first camera image portion, and apply primary PDAF and secondary PDAF to the second camera image portion. Additionally, the processing circuitry is configured to control the camera to focus the first camera image portion and the second camera image portion based on applying the primary PDAF to the first camera image portion and not applying the secondary PDAF to the first camera image portion and based on applying the primary PDAF and the secondary PDAF to the second camera image portion.

[0010] In another example, a method includes identifying a first camera image portion and a second camera image portion in a camera image, applying PDAF to the first camera image portion, and applying primary PDAF and secondary PDAF to the second camera image portion. The method also includes controlling the camera to focus the first camera image portion and the second camera image portion based on applying the primary PDAF to the first camera image portion and not applying the secondary PDAF to the first camera image portion and based on applying the primary PDAF and the secondary PDAF to the second camera image portion.

[0011] In another example, a computer-readable medium stores instructions that, when applied by processing circuitry, cause the processing circuitry to identify a first camera image portion and a second camera image portion in a camera image, apply primary PDAF to the first camera image portion, and apply primary PDAF and secondary PDAF to the second camera image portion; and the instructions also cause the processing circuitry to control the camera to focus the first camera image portion and the second camera image portion based on applying the primary PDAF to the first camera image portion and not applying the secondary PDAF to the first camera image portion and based on applying the primary PDAF and the secondary PDAF to the second camera image portion.

[0012] This Summary is intended to provide an overview of the subject matter described in this disclosure. It is not intended to provide an exclusive or exhaustive explanation of the systems, devices, and methods described in detail within the accompanying drawings and the following description. Further details of one or more examples of this disclosure are set forth in the accompanying drawings and in the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a block diagram of a camera system configured to perform one or more of the example techniques described in this disclosure.

[0014] Figure 2 is a block diagram of a camera, a camera processor, a central processing unit (CPU), and a system memory of the camera system Figure 1 illustrating in more detail

[0015] Figure 3A is a conceptual diagram illustrating a first camera image including a first object and a first image region according to one or more techniques of this disclosure.

[0016] Figure 3B is a conceptual diagram illustrating a second camera image including a second object and a second image region according to one or more techniques of this disclosure.

[0017] Figure 4Ais a conceptual diagram illustrating a long exposure component of a camera image including a region of interest (ROI) in accordance with one or more techniques of this disclosure.

[0018] Figure 4B is a conceptual diagram illustrating a short exposure component of a camera image including a ROI in accordance with one or more techniques of this disclosure.

[0019] Figure 5 is a flowchart of an example method for processing a camera image to focus a camera in accordance with one or more techniques of this disclosure. DETAILED DESCRIPTION

[0020] The example techniques described in this disclosure relate to camera focus (e.g., a depth at which a camera should be focused for image or video capture). Phase-detection autofocus (PDAF) image processing is a technique used to achieve fast and accurate autofocus when capturing in-focus features of a camera image. In some examples, a system can use PDAF when capturing moving objects or complex scenes with depth variations where fast focusing is beneficial. That is, it can be beneficial to reduce the amount of time a system spends using PDAF in order to reduce the amount of time it takes to focus a camera.

[0021] In some examples, a camera can include a lens that refracts light rays and a sensor that generates image data based on the captured light rays. Focusing a camera can involve adjusting the positioning of the lens of the camera relative to the sensor of the camera so that the light rays refracted by the lens converge precisely on the sensor of the camera. When the light rays do not converge in the proper location on the sensor, the resulting camera image can appear out of focus. Out-of-focus images can include blurry objects and / or backgrounds. In some examples, a camera user can manually focus the camera. Additionally or alternatively, a camera can use one or more autofocus techniques to automatically focus the camera without input from the user and based on image data collected by the camera.

[0022] PDAF is an autofocus technique that relies on detecting a phase difference between incident light rays refracted by a lens of a camera and captured by two sets of phase-detection photodiodes. In some examples, the phase difference between incident light rays captured by the two sets of phase-detection photodiodes, respectively, can indicate a degree to which image content is out of focus. The phase difference can additionally or alternatively indicate a direction in which the camera should be focused to bring one or more subjects into focus. For example, the phase difference can indicate whether the lens should be moved closer to the sensor to bring the image into focus or whether the lens should be moved further away from the sensor to bring the image into focus.

[0023] A camera configured for PDAF can include a device for splitting incident light rays into two or more separate paths such that each set of phase detection photodiodes receives a respective light ray. The camera sensor can include phase detection photodiodes (e.g., pixels) configured to detect the phase of the light rays of the two or more separate paths. The camera processor can calculate a phase difference between the light rays of the two or more separate paths. The phase difference can indicate a degree to which the camera is out of focus and a direction in which the positioning of the lens should be adjusted to bring the camera into focus. The camera can adjust the lens relative to the sensor based on the calculated phase difference to bring the camera into focus.

[0024] In some examples, PDAF can be performed horizontally and / or vertically. Horizontal PDAF can bring a camera image into focus in a manner that is sensitive to phase differences that move across the image horizontally. That is, a camera performing horizontal PDAF can bring into focus a scene in which an object having a first depth is separated from a background having a second depth by a substantially vertical border. However, horizontal PDAF can not bring into focus a scene in which an object having a first depth is separated from a background having a second depth by a substantially horizontal border. This is because a processor performing horizontal PDAF can bring into focus more effectively an image having a texture that includes changes along a horizontal axis than an image having a texture that does not include changes along a horizontal axis. A vertical border can introduce changes in image texture along a horizontal axis, and a horizontal border can introduce changes in image texture along a vertical axis.

[0025] In some examples, a camera system can use horizontal PDAF to bring a camera into focus automatically. However, in some cases, horizontal PDAF can not be sufficient to bring one or more portions of a camera image into focus. For one or more portions of a camera image that horizontal PDAF is not able to bring into focus sufficiently, the system can use vertical PDAF to bring one or more regions into focus. In some cases, vertical PDAF can help bring into focus portions of a camera image that horizontal PDAF can not bring into focus effectively. For example, when a region of a camera image includes substantially horizontal texture features that are not easily identified by processing data from left to right or right to left, vertical PDAF can be used to bring the horizontal texture features into focus.

[0026] The camera system described herein can use an "on-demand" PDAF to initially process a camera image using a primary PDAF technique and process portions of the camera image using a secondary PDAF technique when the primary PDAF technique is insufficient to bring one or more portions of the camera image into focus. In some cases, the camera system can use the secondary PDAF technique based on a confidence value associated with the primary PDAF technique. The confidence value can indicate a degree to which the computing system is confident that the primary PDAF technique is sufficient. In some cases, the secondary PDAF technique can use computational resources in addition to those used for the primary PDAF technique. In addition to the time taken to apply the primary PDAF technique, applying the secondary PDAF technique can take time. By using on-demand PDAF, the secondary PDAF technique is applied only when the primary PDAF technique is insufficient to bring one or more regions of the camera image into focus.

[0027] In some examples, the primary PDAF technique can include horizontal PDAF and the secondary PDAF technique can include vertical PDAF. That is, the camera system can process a camera image using horizontal PDAF and process one or more regions of the camera image using vertical PDAF when the horizontal PDAF is insufficient to bring the region(s) into focus. This can allow the camera system to reserve the use of vertical PDAF to only when needed to bring certain regions of the image into focus, thus reducing the amount of time needed to bring the image into focus. In some examples, the vertical PDAF can be performed using a hardware unit, software, or a combination of hardware and software.

[0028] In some examples, a camera image can be captured using multiple exposures. That is, a long exposure component and a short exposure component can be combined to create a camera image. In some examples, the system can use hardware to perform a primary PDAF technique on the long exposure component of the camera image. When needed, the system can further process the short exposure component of the camera image according to a secondary PDAF technique using software. This is because using hardware to perform PDAF on the long exposure component can produce insufficient results in some cases, and further processing the short exposure component can improve the focused image. Additionally or alternatively, the system can use hardware to perform a primary PDAF technique on the short exposure component of the camera image. When needed, the system can further process the long exposure component of the camera image according to a secondary PDAF technique using software.

[0029] Figure 1is a block diagram of a camera system 10 configured to perform one or more of the example techniques described in this disclosure. Examples of camera system 10 include a standalone digital camera or digital video camcorder, a wireless communication device handset equipped with a camera such as a mobile telephone having one or more cameras, a cellular or satellite radio telephone, a personal digital assistant (PDA) equipped with a camera, a computing panel or tablet, a watch, a gaming device, a computer device including a camera such as a so-called "webcam," or any device having digital imaging or video capabilities.

[0030] As Figure 1 illustrated by the example of Figure 1 Although the example of

[0031] Furthermore, although the various components are illustrated as separate components, in some examples, these components can be combined to form a system on a chip (SoC). As one example, camera processor 14, CPU 16, GPU 18, and display interface 26 can be formed on a common integrated circuit (IC) chip. In some examples, one or more of camera processor 14, CPU 16, GPU 18, and display interface 26 can be located in separate IC chips. Additional examples of components that can be configured to perform example techniques include a digital signal processor (DSP), a vector processor, or other hardware blocks for neural network (NN) computations. Various other arrangements and combinations are possible, and the techniques should not be considered limited to the examples illustrated in Figure 1

[0032] In Figure 1 ​The various components illustrated in FIG. 1 can be formed as at least one of fixed function or programmable circuitry, such as in one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), or other equivalent integrated or discrete logic circuitry. Examples of local memory 20 and system memory 30 include one or more volatile or non-volatile memories or storage devices, such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, a magnetic data medium, or an optical storage medium.

[0033] In Figure 1 The various units illustrated in FIG. 1 communicate with one another using bus 32. Bus 32 can be any of various bus structures, such as a third-generation bus (e.g., a HyperTransport bus or an InfiniBand bus), a second-generation bus (e.g., an advanced graphics port bus, a peripheral component interconnect (PCI) express bus, or an advanced extensible interface (AXI) bus), or another type of bus or device interconnect. In Figure 1 The specific configuration of buses and communication interfaces between the different components shown in FIG. 1 is merely exemplary, and other configurations of camera devices and / or other image processing systems having the same or different components can be used to implement the techniques of this disclosure.

[0034] In some examples, camera 12 can include a lens and a sensor. The lens can refract light onto the sensor, which generates one or more image frames for output. In some examples, the sensor corresponding to camera 12 can include image pixels that capture image data and phase detection pixels for autofocus techniques such as PDAF. PDAF is a technique used to achieve fast and accurate focus when capturing clear images. For example, when one or more objects are moving in a 3D environment, or when one or more objects are located between the camera and the background, a camera using PDAF can focus on the objects more quickly and efficiently than objects that do not use PDAF to focus image data.

[0035] The sensor of the camera 12 can include dedicated phase pixels separate from image pixels of the sensor of the camera 12. The camera 12 can direct light to both the phase pixels and the image pixels. The image pixels can capture image data that forms a camera image. In some examples, the phase pixels can include pairs of pixels (e.g., “left” and “right” pairs of pixels and / or “up” and “down” pairs of pixels). The phase pixels can perform phase detection by detecting a phase difference between the pixels of each pair of pixels. The phase pixels can be part of the camera sensor hardware. A PDAF hardware unit for processing data collected by the camera 12 can be separate from the hardware of the camera 12.

[0036] The camera system 10 can use hardware to perform PDAF. For example, a PDAF hardware unit can process data captured by the phase pixels and / or image pixels of the sensor of the camera 12 to perform PDAF. The camera system 10 can additionally or alternatively use software to perform PDAF. For example, the system memory 30 can be configured to store PDAF software configured to process data collected by the sensor of the camera 12 to perform PDAF. In some examples, the system memory 30 can be configured to store one or more models (e.g., machine learning models, neural networks) configured to process data collected by the sensor of the camera 12 to perform PDAF. In some examples, the camera system 10 uses software to perform PDAF in addition to using hardware to perform PDAF.

[0037] In some examples, using software to perform PDAF consumes a greater amount of computing resources and / or consumes a greater amount of energy than using dedicated hardware components (e.g., phase detection pixels and a PDAF hardware unit) to perform PDAF without using software. In some examples, using software to perform PDAF takes a greater amount of time than using dedicated hardware components (e.g., phase detection pixels and a PDAF hardware unit) to perform PDAF without using software. Because it can be beneficial to perform PDAF quickly while conserving energy, in some examples, the computing device 10 can use hardware to perform PDAF and use software as a supplement to the hardware only when the hardware is insufficient for autofocus.

[0038] Camera 12 can be configured to perform horizontal PDAF and / or vertical PDAF. When camera 12 is configured for horizontal PDAF, phase detection pixels can be arranged horizontally across the sensor of camera 12. In some examples, this means that a pair of phase detection pixels can include a "left" pixel and a "right" pixel arranged horizontally. Horizontally arranged phase detection pixels can be sensitive to horizontal phase differences between light rays from different parts of the lens of camera 12. When camera 12 is configured for vertical PDAF, phase detection pixels can be arranged vertically across the sensor of camera 12. In some examples, this means that a pair of phase detection pixels can include an "up" pixel and a "down" pixel arranged vertically. Vertically arranged phase detection pixels can be sensitive to vertical phase differences between light rays from different parts of the lens of camera 12.

[0039] In some examples, when camera system 10 performs horizontal PDAF, camera system 10 can effectively focus on objects having a substantially vertical line and / or a substantially diagonal line with respect to the camera sensor. Because camera system 10 processes data horizontally when performing horizontal PDAF, camera system 10 can be configured to easily identify vertical texture features because the phase can change when moving horizontally across a vertical texture feature. In some examples, when camera system 10 performs vertical PDAF, camera system 10 can effectively focus on objects having a substantially horizontal line and / or a substantially diagonal line with respect to the camera sensor. Because camera system 10 processes data vertically when performing horizontal PDAF, camera system 10 can be configured to easily identify horizontal texture features because the phase can change when moving vertically across a horizontal texture feature.

[0040] In some examples, camera system 10 can use horizontal PDAF to process camera images collected by camera 12. In some examples, horizontal PDAF is sufficient to process one or more regions of a camera image. But in some cases (e.g., where a region of a camera image includes one or more substantially horizontal lines), camera system 10 can use vertical PDAF processing to achieve sufficient results. In some examples, camera system 10 can use hardware to perform horizontal PDAF and use software to perform vertical PDAF when horizontal PDAF is insufficient to process a region of a camera image. In some examples, camera system 10 can use hardware to perform horizontal PDAF and use software to perform vertical PDAF when horizontal PDAF is insufficient to process a region of a camera image.

[0041] The camera processor 14 is configured to receive image frames (or simply“images” or“camera images”) from the camera 12 and process the images to generate output images for display. The CPU 16, GPU 18, camera processor 14, or some other circuitry can be configured to process the output images, including image content generated by the camera processor 14, into images for display on the display 28. In some examples, the GPU 18 can also be configured to render graphics content on the display 28.

[0042] In some examples, the camera processor 14 can be configured as an image processing pipeline. For example, the camera processor 14 can include a camera interface that interfaces between the camera 12 and the camera processor 14. The camera processor 14 can include additional circuitry for processing image content. The camera processor 14 outputs the resulting image with image content (e.g., pixel values for each of the image pixels) to the system memory 30 via the memory controller 24. In some examples, the camera processor 14 can include one or more PDAF hardware units configured to process image data and / or phase data collected by the sensor of the camera 12. For example, the camera processor 14 can include a horizontal PDAF hardware unit for performing horizontal PDAF and / or a vertical PDAF hardware unit for performing vertical PDAF.

[0043] In some examples, the camera processor 14 can receive one or more camera images from the camera 12 for processing. Additionally or alternatively, the camera processor 14 can receive one or more camera images from the system memory 30 for processing. The camera processor 14 can be configured to identify, for each of the one or more camera images, a first camera image portion and a second camera image portion. In some examples, the camera processor 14 can be configured to perform PDAF on the first portion of the camera image using horizontal PDAF without using vertical PDAF and / or using hardware without using software. In some examples, the camera processor 14 can not be configured to perform PDAF on the second portion of the camera image using horizontal PDAF without using vertical PDAF and / or using hardware without using software.

[0044] In some examples, camera processor 14 can apply horizontal PDAF to the first camera image portion to generate the first processed camera image portion. Applying horizontal PDAF without applying vertical PDAF can be sufficient to generate the first processed camera image portion that exceeds the threshold level of PDAF quality. That is, camera processor 14 can sufficiently process the first camera image portion by applying horizontal PDAF without needing to use additional computational resources and consume additional power to apply vertical PDAF to the first camera image portion. Camera processor 14 can sufficiently process the first camera image portion by applying horizontal PDAF without expending additional time to apply vertical PDAF to the first camera image portion.

[0045] In some examples, camera processor 14 can apply a PDAF hardware unit to the first camera image portion to generate the first processed camera image portion. Applying the PDAF hardware unit without applying software stored by system memory 30 can be sufficient to generate the first processed camera image portion that exceeds the threshold level of PDAF quality. That is, camera processor 14 can sufficiently process the first camera image portion by applying the PDAF hardware unit without needing to use additional computational resources and consume additional power to apply software to the first camera image portion. Camera processor 14 can sufficiently process the first camera image portion by applying the PDAF hardware unit without expending additional time to apply software stored by system memory 30 to the first camera image portion.

[0046] In some cases, camera processor 14 can apply both horizontal PDAF and vertical PDAF to the second camera image portion to generate the second processed camera image portion. That is, camera processor 14 can use vertical PDAF to process the second camera image portion only when using horizontal PDAF fails to produce sufficient results. In some cases, camera processor 14 can use both hardware of camera processor 14 and software stored by system memory 30 for the second camera image portion to apply PDAF to generate the second processed camera image portion. That is, camera processor 14 can use software to process the second camera image portion only when using hardware fails to produce sufficient results.

[0047] CPU 16 can include a general or special purpose processor that controls operation of the camera system 10. A user can provide input to the camera system 10 to cause the CPU 16 to execute one or more software applications. The software applications executing on the CPU 16 can include, for example, a media player application, a video game application, a graphical user interface application, or another program. The user can provide input to the camera system 10 via one or more input devices (not shown), such as a keyboard, a mouse, a microphone, a touchpad, or another input device coupled to the camera system 10 via the user interface 22.

[0048] One example of a software application is a camera application. The CPU 16 executes the camera application, and in response, the camera application causes the CPU 16 to generate content that the display 28 outputs. The GPU 18 can be configured to process the content generated by the CPU 16 for rendering on the display 28. For example, the display 28 can output information such as light intensity, whether a flash is enabled, and other such information. A user of the camera system 10 can interface with the display 28 to configure the manner in which images are generated (e.g., with or without a flash, focus settings, exposure settings, and other parameters).

[0049] As one example, after the camera application is executed, the camera system 10 can be considered to be in a preview mode. In the preview mode, the camera 12 outputs image content to the camera processor 14, which performs camera processing and outputs the image content to the system memory 30, which the display interface 26 retrieves and outputs on the display 28. In the preview mode, a user can view, via the display 28, the image content that will be captured when the user presses a button (either a real button or a button on the display) to take a picture. As another example, a user can record video content (e.g., a series of images) rather than take a still image (e.g., a picture). During recording, the user can be able to view the image content being captured on the display 28.

[0050] In this disclosure, a preview image can be referred to as an image generated in a preview mode. For example, in the preview mode, an image that the camera 12 outputs and stores (e.g., in the local memory 20 or the system memory 30) for processing by the camera processor 14 or that the camera processor 14 generates and stores (e.g., in the local memory 20 or the system memory 30) can be referred to as a preview image. In general, a preview image can be an image generated in a preview mode prior to capture and long-term storage of the image.

[0051] During preview mode or recording, camera system 10 (e.g., via CPU 16) can control the way camera 12 captures images (e.g., before capturing or storing images). The present disclosure describes example techniques as being performed by CPU 16. However, the example techniques should not be considered as limited to CPU 16 performing the example techniques. For example, CPU 16 in combination with camera processor 14, GPU 18, DSP, vector processor, and / or display interface 26 can be configured to perform the example techniques described in the present disclosure. For example, one or more processors can be configured to perform the example techniques described in the present disclosure. Examples of one or more processors include camera processor 14, CPU 16, GPU 18, display interface 26, DSP, vector processor, or any combination of one or more of camera processor 14, CPU 16, GPU 18, display interface 26, DSP, or vector processor.

[0052] CPU 16 can be configured to control exposure and / or focus to capture visually pleasing images. For example, CPU 16 can be configured to generate signals that control exposure, focus, and white balance (as a few non-limiting examples) of camera 12. CPU 16 can be configured to control exposure, focus, and white balance based on preview images received from camera processor 14 during preview mode or recording. In this way, for still images, exposure, focus, and white balance are adjusted (e.g., parameters for exposure, focus, and possibly white balance are determined prior to image capture so that exposure, focus, and white balance can be corrected during image capture) as the user engages in taking a picture. For recording, exposure, focus, and white balance can be updated periodically during recording.

[0053] Memory controller 24 facilitates the transfer of data into and out of system memory 30. For example, memory controller 24 can receive memory read and write commands and service such commands with respect to system memory 30 in order to provide memory services for components in camera system 10. Memory controller 24 is communicatively coupled to system memory 30. Although memory controller 24 is illustrated in the example of camera system 10 as being separate processing circuitry from both CPU 16 and system memory 30, in other examples, some or all of the functionality of memory controller 24 can be implemented on one or both of CPU 16 and system memory 30. Figure 1 Although memory controller 24 is illustrated in the example of camera system 10 as being separate processing circuitry from both CPU 16 and system memory 30, in other examples, some or all of the functionality of memory controller 24 can be implemented on one or both of CPU 16 and system memory 30.

[0054] System memory 30 can store program modules and / or instructions and / or data accessible by camera processor 14, CPU 16, and GPU 18. For example, system memory 30 can store user applications (e.g., instructions for a camera application), resulting frames from camera processor 14, etc. System memory 30 can additionally store information for use and / or generation by other components of camera system 10. For example, system memory 30 can serve as device memory for camera processor 14.

[0055] Figure 2 is a block diagram of camera 12, camera processor 14, CPU 16, and system memory 30 of a camera system that is illustrated in more detail Figure 1 . As illustrated Figure 2 , camera 12 includes lens 34 and sensor 36. Sensor 36 includes image pixels 42, horizontal phase pixels 44, and vertical phase pixels 46. Camera processor 14 includes PDAF hardware 47, which includes horizontal PDAF hardware unit 48 and vertical PDAF hardware unit 50. In some examples, PDAF hardware 47 does not include vertical PDAF hardware unit 50 and includes horizontal PDAF hardware unit 48. System memory 30 is configured to store image data 52, vertical PDAF software 54, and model 58.

[0056] Camera 12 can be configured to generate one or more camera images. For example, lens 34 of camera 12 can refract light onto image pixels 42 of sensor 36. Image pixels 42 can be configured to generate data (e.g., digital pixel data) that includes one or more camera images. In some examples, digital pixel data can include color data and / or intensity data corresponding to each of image pixels 42. In some cases, sensor 36 of camera 12 can be configured to output one or more camera images to processor 14. In some examples, sensor 36 can be configured to output one or more camera images to system memory 30 for storage as part of image data 52.

[0057] In some examples, the sensor 36 can be configured to generate phase data corresponding to each of the one or more camera images. For example, the sensor 36 can include two or more sets of photodiodes, and a phase difference in image content captured by different sets of photodiodes is an example of phase data. The phase data can be used to autofocus the camera. That is, based on the phase data generated by the sensor 36, the camera system 10 can control a focus of the lens 34 so as to bring one or more objects in the image data generated by the image pixels 42 into focus. In some examples, to control the focus of the lens 34, the camera system 10 can control a distance between the lens 34 and the sensor 36, control an angle of the lens 34 relative to the sensor 36, or any combination thereof.

[0058] The sensor 36 can include horizontal phase pixels 44. In some examples, the horizontal phase pixels 44 can be arranged horizontally across the sensor 36. In some examples, the horizontal phase pixels 44 can include one or more pairs of “left” and “right” pixels. That is, each pair of horizontal phase pixels can include a side-by-side pair including a left phase pixel and a right phase pixel. One or more pairs of pixels in the horizontal phase pixels 44 can be configured to detect horizontal phase differences between light rays refracted from different locations on the lens 34. These phase differences can be indicative of differences between objects within the 3D environment. Using the phase data generated by the horizontal phase pixels 44, the camera system 10 can control a focus of the camera 12 so that objects within the 3D environment appear sharp and in focus in the camera images generated by the camera 12.

[0059] The sensor 36 can include vertical phase pixels 46. In some examples, the vertical phase pixels 46 can be arranged vertically across the sensor 36. In some examples, the vertical phase pixels 46 can include one or more pairs of “up” and “down” pixels. That is, each pair of vertical phase pixels can include a pair with a first pixel above another pixel. One or more pairs of pixels in the vertical phase pixels 46 can be configured to detect vertical phase differences between light rays refracted from different locations on the lens 34. These phase differences can be indicative of differences between objects within the 3D environment. Using the phase data generated by the vertical phase pixels 46, the camera system 10 can control a focus of the camera 12 so that objects within the 3D environment appear sharp and in focus in the camera images generated by the camera 12.

[0060] In some examples, the one or more camera images generated by the camera 12 can include image data generated by the image pixels 42, phase data generated by the horizontal phase pixels 44, phase data generated by the vertical phase pixels 46, or any combination thereof. The camera processor 14 can be configured to process the one or more camera images to autofocus the camera 12 (e.g., apply PDAF). In some examples, autofocusing the camera 12 can be beneficial to enable the camera 12 to capture high-quality images of one or more objects within a 3D environment. In some examples, the focal point of the camera 12 to capture high-quality images can depend on a number of factors, including the location of the one or more objects relative to the camera 12, the location of the one or more objects relative to the background, the light level of the 3D environment, or any combination thereof.

[0061] In some examples, the camera processor 14 can be configured to use the horizontal PDAF hardware unit 48 to process phase data collected by the horizontal phase pixels 44 and / or the image pixels 42 in order to autofocus the camera 12. That is, the camera processor 14 can apply the horizontal PDAF hardware unit 48 to data generated by the horizontal phase pixels 44 and / or the image pixels 42 to generate an output. In some examples, using the horizontal PDAF hardware unit 48 can be sufficient to autofocus the camera 12. In some examples, using the horizontal PDAF hardware unit 48 can not be sufficient to autofocus the camera 12.

[0062] In some examples, the camera processor 14 can identify a first camera image portion and a second camera image portion of a camera image generated by the camera 12. In some examples, the camera processor 14 can identify the first camera image portion as a portion of the camera image that the camera processor 14 is configured to adequately process using the horizontal PDAF hardware unit 48 (e.g., using the horizontal PDAF hardware unit 48 can enable adequately accurate autofocus). In some examples, the camera processor 14 can identify the first camera image portion as a portion of the camera image that the camera processor 14 is configured to adequately process using any PDAF hardware unit. In some examples, the camera processor 14 can identify the second camera image portion as a portion of the camera image that the camera processor 14 is not configured to adequately process using the horizontal PDAF hardware unit 48 (e.g., using the horizontal PDAF hardware unit 48 can not enable adequately accurate autofocus). In some examples, the camera processor 14 can identify the second camera image portion as a portion of the camera image that the camera processor 14 is not configured to adequately process using any PDAF hardware unit 48.

[0063] In some examples, to identify the first camera image portion and the second camera image portion of the camera image generated by the camera 12, the camera processor 14 is configured to determine a set of confidence scores. Each confidence score of the set of confidence scores corresponds to an image region of the set of image regions of the camera image generated by the camera 12. That is, the camera processor 14 can generate a confidence score corresponding to each image region of the set of image regions of the camera image. In some examples, the confidence score corresponding to each image region of the set of image regions can represent a confidence that the horizontal PDAF hardware unit 48 is configured to adequately process image data corresponding to the respective image region.

[0064] In some cases, the camera processor 14 can compare each confidence score of the set of confidence scores to a threshold confidence score. In some examples, the camera processor 14 can identify the first camera image portion to include each image region of the set of image regions corresponding to a confidence score that is greater than the threshold confidence score. That is, the first camera image portion can include regions associated with a high confidence that the region is processed using the horizontal PDAF hardware unit 48 and / or any PDAF hardware unit to achieve adequate results. The camera processor 14 can identify the second camera image portion to include each image region of the set of image regions corresponding to a confidence score that is not greater than the threshold confidence score. That is, the first camera image portion can include regions associated with a low confidence that the region is processed using the horizontal PDAF hardware unit 48 and / or any PDAF hardware unit to achieve adequate results.

[0065] To determine the confidence score corresponding to each image region of the set of image regions, the camera processor 14 can identify a sum of absolute differences (SAD) curve corresponding to each image region of the set of image regions. In some examples, the camera processor 14 can calculate a SAD corresponding to each image region of the set of image regions for each offset value of a set of offset values. For example, the camera processor 14 can calculate a SAD corresponding to an image region and an image region offset by n pixels, calculate a SAD corresponding to an image region and an image region offset by n-1 pixels, etc. The camera processor 14 can determine each confidence score of the set of confidence scores based on the SAD curve corresponding to the respective image region. In some examples, the processor 14 can identify the SAD curve based on pixel data (e.g., color data, intensity data).

[0066] To determine each confidence score in the set of confidence scores based on the SAD curve corresponding to the respective image region, the camera processor 14 is configured to identify a minimum value of the SAD curve corresponding to each image region in the set of image regions and to compute an average value of the SAD curve corresponding to each image region in the set of image regions. The camera processor 14 can compute each confidence score in the set of confidence scores based on the minimum value of the SAD curve corresponding to the respective image region and based on the average value of the SAD curve corresponding to the respective image region.

[0067] For example, the camera processor 14 can compute each confidence score in the set of confidence scores as a difference between the minimum value of the SAD curve corresponding to the respective image region and the average value of that SAD curve. In another example, the camera processor 14 can compute each confidence score in the set of confidence scores as a ratio of the average value of the SAD curve corresponding to the respective image region and the minimum value of that SAD curve. In any case, the confidence score corresponding to the image region associated with the SAD curve can be lower when the minimum value of the SAD curve is closer to the average value of the SAD curve than when the minimum value of the SAD curve is further from the average value of the SAD curve.

[0068] In some examples, the camera processor 14 can use the horizontal PDAF hardware unit 48 to process the first camera image portion identified by the camera processor 14 and use the horizontal PDAF hardware unit 48 to process the second camera image portion identified by the camera processor 14. This means that the camera processor 14 can use the horizontal PDAF hardware unit 48 to process both the first camera image portion and the second camera image portion even though the horizontal PDAF hardware unit 48 can be sufficient to process the first camera portion and can not be sufficient to process the second camera portion. However, because the horizontal PDAF hardware unit 48 can not be sufficient to process the second camera portion, the camera processor 14 can supplement the hardware-based horizontal PDAF processing to process the second camera portion with one or more other kinds of PDAF processing.

[0069] For example, the camera processor 14 can use the vertical PDAF software 54 stored by the system memory 30 to process the second camera portion. In some examples, the camera processor 14 can use the vertical PDAF software 54 stored by the system memory 30 to process the second camera portion in addition to using the horizontal PDAF hardware unit 48 to process the second camera portion. In some examples, the camera processor 14 can use the vertical PDAF software 54 stored by the system memory 30 to process the second camera portion instead of using the horizontal PDAF hardware unit 48 to process the second camera portion.

[0070] In some examples, processing the second camera image portion using both the horizontal PDAF hardware unit 48 and the vertical PDAF software 54 can consume a greater amount of computational resources and / or take a greater amount of time compared to processing the second camera image portion using the horizontal PDAF hardware unit 48 without using the horizontal PDAF hardware unit 48. In some examples, processing the second camera image portion using the vertical PDAF software 54 can consume a greater amount of computational resources and / or take a greater amount of time compared to processing the second camera image portion using the horizontal PDAF hardware unit 48. This means that processing the second camera image portion using the horizontal PDAF hardware unit 48 without using the vertical PDAF software 54 can be beneficial.

[0071] In some cases, the camera processor 14 can process the second camera portion using the model 58 stored by the system memory 30. For example, the camera processor 14 can process the camera image portion using a vertical PDAF deep neural network (DNN). In some examples, the camera processor 14 can process the second camera portion using the model 58 stored by the system memory 30 in addition to processing the second camera portion using the horizontal PDAF hardware unit 48. In some examples, the camera processor 14 can process the second camera portion using the model 58 stored by the system memory 30 instead of processing the second camera portion using the horizontal PDAF hardware unit 48.

[0072] The camera processor 14 can be configured to store a vertical PDAF hardware unit 50. In some examples, the vertical PDAF hardware unit 50 can process a camera image output by the camera 12. For example, the vertical PDAF hardware unit 50 can process image data output from the image pixels 42, phase data output from the horizontal phase pixels 44, phase data output from the vertical phase pixels 46, or any combination thereof. In some examples, the camera processor 14 can use the vertical PDAF hardware unit 50 to process one or more portions of a camera image that the horizontal PDAF hardware unit 48 is unable to adequately process. For example, the camera processor 14 can use the horizontal PDAF hardware unit 48 to process a first camera image portion and a second camera image portion. When the results of processing the second camera image portion using the horizontal PDAF hardware unit 48 do not exceed a threshold level of quality, the camera processor 14 can use the vertical PDAF hardware unit 50 to process the second camera image portion.

[0073] In some examples, camera 12 can capture one or more camera images using multiple exposures. In some examples, multiple exposures involve capturing multiple images using the same sensor to create a composite image. Some multiple exposure techniques involve capturing a long exposure component and a short exposure component and combining the long exposure component and the short exposure component into one camera image. For example, sensor 36 can capture a long exposure component and a short exposure component and generate a camera image based on the long exposure component and the short exposure component. In some examples, the pixels 42 are exposed to light for a longer period of time for the long exposure component image than for the short exposure component image.

[0074] In some examples, camera processor 14 can use PDAF hardware 47 to process the long exposure component of a camera image. In some examples, camera processor 14 can use horizontal PDAF hardware unit 48, vertical PDAF hardware unit 50, or both horizontal PDAF hardware unit 48 and vertical PDAF hardware unit 50 to process the long exposure component of a camera image. In some examples, camera processor 14 can evaluate the long exposure of a camera image to determine whether PDAF hardware 47 is sufficient to process the long exposure component. When PDAF hardware 47 is not sufficient to process the long exposure component, camera processor 14 can use software (e.g., vertical PDAF software 54) to process the short exposure component.

[0075] In some examples, to determine whether PDAF hardware 47 is sufficient to process the long exposure component, camera processor 14 can determine whether a region of interest (ROI) of the long exposure component satisfies one or more criteria. For example, processor 14 can determine a confidence score corresponding to the ROI of the long exposure component and determine whether the confidence score of the ROI is greater than a threshold confidence score. When the confidence score of the ROI is greater than the threshold confidence score, camera processor 14 can determine that the ROI of the long exposure component satisfies the one or more criteria. When the confidence score of the ROI is not greater than the threshold confidence score, camera processor 14 can determine that the ROI of the long exposure component does not satisfy the one or more criteria.

[0076] To determine whether PDAF hardware 47 is sufficient to process the long exposure component, in some examples, camera processor 14 can determine an exposure level of the ROI of the long exposure component and determine whether the exposure level of the ROI is greater than a threshold exposure level. When the exposure level of the ROI is greater than the threshold exposure level, camera processor 14 can determine that the ROI of the long exposure component satisfies the one or more criteria. When the exposure level of the ROI is not greater than the threshold exposure level, camera processor 14 can determine that the ROI of the long exposure component does not satisfy the one or more criteria.

[0077] In any case where the long-exposure component including the ROI does not satisfy one or more criteria (e.g., when the ROI of the long-exposure component does not have a high confidence, or when the ROI of the long-exposure component is overexposed), the camera processor 14 can process the short-exposure component of the camera image with software. For example, when the ROI of the long-exposure component does not satisfy one or more criteria, the camera processor 14 can process the short-exposure component with software stored in the system memory 30 (e.g., vertical PDAF software 54, model 58, or other software stored in the system memory 30).

[0078] By processing the short-exposure component of the camera image with software only when the ROI of the long-exposure component does not satisfy one or more criteria, the camera processor 14 can conserve computational resources and reduce processing latency compared to systems that process the long-exposure component and both the long-exposure component and the short-exposure component in every case. The camera processor 14 can also ensure that the camera image is adequately processed without using resources and spending additional time to process with software when software processing is not needed.

[0079] Figure 3A is a conceptual diagram illustrating a first camera image 60 including a first object 61 and a first image region 62 in accordance with one or more techniques of the present disclosure. As Figure 3A seen, the first object 61 is a triangular object having a set of object segments that are arranged vertically on top of one another. For example, the set of object segments of the first object 61 includes one or more gaps between the set of object segments. The first object 61 is presented against a background. The first camera image 60 includes one or more texture features corresponding to the first object 61 and the background. For example, texture feature 64 includes a diagonal boundary between the first object 61 and the background, and texture feature 66 includes a horizontal boundary between the first object 61 and the background. The first object 61 can include one or more regions of the first camera image 60 for which texture of the first camera image 60 results in inaccurate vertical PDAF processing.

[0080] In some examples, the first object 61 can be associated with a low horizontal PDAF confidence value. This is because there are horizontal texture features including the texture feature 66 between the first object 61 and the background, which are difficult to process using horizontal PDAF. For example, when the first image region 62 is offset by one pixel, the texture feature 66 appears to be the same in the offset region. This means that the minimum value of the SAD curve corresponding to the first image region 62 can not be very different from the average value of the SAD curve corresponding to the first image region 62, and thus the first image region 62 can be associated with a low horizontal PDAF confidence value.

[0081] Because texture feature 64 is diagonal rather than horizontal, horizontal PDAF can effectively handle the portion of the first image region 62 that includes diagonal texture features (including texture feature 64). However, because the first object 61 includes several horizontal texture features (including texture feature 66), the first object 61 may generally be associated with low horizontal PDAF confidence. This means that it may be beneficial to use vertical PDAF (e.g., hardware PDAF and / or software PDAF) to process one or more regions of the first camera image 60 to make the horizontal texture features between the first object 61 and the background more fully focused.

[0082] Hardware processing for vertical PDAF (e.g., using) Figure 2 The vertical PDAF hardware unit 50 may require a significant area due to the row buffer requirements used for registration. Strong horizontal merging can significantly reduce the data rate and apply uniformity correction. When the horizontal PDAF confidence is low, it may be beneficial for the camera system 10 to perform vertical processing in some areas (e.g., the first image region 62). The latency associated with several PDAF modes is presented in the following table.

[0083]

[0084] Table 1

[0085] As shown in Table 1, processing camera images using the horizontal PDAF hardware unit 48 takes less time compared to using both the horizontal PDAF hardware unit 48 and the vertical PDAF software 54. Processing camera images using the horizontal PDAF hardware unit 48 also takes less time compared to using both the horizontal PDAF hardware unit 48 and the vertical PDAF hardware unit 50. However, when the horizontal PDAF is insufficient, processing a region of the camera image using on-demand vertical PDAF takes less time compared to processing the entire image using software and / or hardware vertical PDAF. This means that using vertical PDAF on demand as a supplement to horizontal PDAF can be beneficial.

[0086] Figure 3B This is a conceptual diagram illustrating a second camera image 70 including a second object 71 and a second image region 72, according to one or more techniques of this disclosure. Figure 3BAs seen, the second object 71 is a representation of an upper torso and head of a human subject. The second image region 72 can be centered on the face of the human subject. In some examples, the second image region 72 includes one or more curved texture features. These curved texture features include one or more curved texture features between the hairline of the human subject and the face of the subject and one or more curved texture features between the face of the patient and the neck of the patient. These curved texture features can include features that are substantially vertically aligned, features that are substantially horizontally aligned, and features that are substantially diagonally aligned.

[0087] In some examples, the second image region 72 can be associated with a horizontal PDAF confidence that is higher than the horizontal PDAF confidence of the first image region 62 of Figure 3A Although some texture features within the second image region 72 (e.g., the chin of the human subject) can be more difficult to process using horizontal PDAF than other texture features, the features within the second image region 72 are generally less horizontal than the features within the first image region 62 of Figure 3A This means that horizontal PDAF can be more effective for processing the second image region 72 than the effectiveness of using horizontal PDAF to process the first image region 62 of Figure 3A Accordingly, it can be sufficient for the camera system 10 to process the second image region 72 using the horizontal PDAF hardware unit 48 without using vertical PDAF.

[0088] Figure 4A is a conceptual diagram illustrating a long-exposure component 80 of a camera image that includes an ROI 82 in accordance with one or more techniques of the present disclosure. As Figure 4A seen, the long-exposure component 80 includes two human subjects characterized in the ROI 82 relative to a background 84. Because the sensor that generates the long-exposure component 80 can be exposed to light for a greater amount of time than a short-exposure component, the background of the long-exposure component 80 can appear bright or completely white.

[0089] Figure 4B is a conceptual diagram illustrating a short-exposure component 90 of a camera image that includes an ROI 92 in accordance with one or more techniques of the present disclosure. In some examples, the short-exposure component 90 and the long-exposure component 80 can represent components of the same camera image. As Figure 4B seen, the short-exposure component 90 includes two human subjects characterized in the ROI 92 relative to a background 94. Because the sensor that generates the short-exposure component 90 can be exposed to light for a lesser amount of time than the long-exposure component 80, the background 94 and the human subjects of the short-exposure component 90 can appear darker than the background 84 and the human subjects of the long-exposure component 80.

[0090] The camera system 10 can use the PDAF hardware 47 to process the long exposure component 80. In some examples, when the PDAF confidence of the long exposure component 80 is not greater than a confidence threshold, the camera system 10 can use software stored by the system memory 30 to process the short exposure component 90. In some examples, when the exposure level of the ROI 82 of the long exposure component 80 is greater than a threshold exposure level, the camera system 10 can use software stored by the system memory 30 to process the short exposure component 90. In some examples, applying PDAF processing to both the long exposure component 80 and the short exposure component 90 can take a greater amount of computational resources and / or a greater amount of time than applying PDAF processing to only the long exposure component 80. This means that it can be beneficial to use on-demand PDAF processing to apply PDAF processing to the short exposure component 90 when needed, but it is not beneficial to apply PDAF processing to the short exposure component 90 when not needed.

[0091] Figure 5 is a flowchart of an example method for processing a camera image to focus a camera, illustrating one or more techniques in accordance with the present disclosure. Figure 5 is described with respect to Figure 1 and Figure 2 the camera system 10, the camera 12, the camera processor 14, the CPU 16, and the system memory 30. However, Figure 5 techniques of may be performed by different components of the camera system 10, the camera 12, the camera processor 14, the CPU 16, and the system memory 30, or by additional or alternative systems.

[0092] The camera processor 14 can identify a first camera image portion and a second camera image portion in the camera image (102). The camera processor 14 can apply primary PDAF to the first camera image portion (104). The camera processor 14 can apply primary PDAF and secondary PDAF to the second camera image portion (106). In some examples, applying primary PDAF without applying secondary PDAF is sufficient to control the camera to focus the first-processed camera image portion. In some examples, applying primary PDAF without applying secondary PDAF is not sufficient to control the camera to focus the second-processed camera image portion. To determine whether applying primary PDAF is sufficient to control the camera to focus one or more portions of the camera image, the camera processor 14 can identify one or more confidence values corresponding to the primary PDAF. In some cases, the primary PDAF is sufficient to control the camera to focus both the first camera image portion and the second camera image portion, and no secondary PDAF is needed to focus the first camera image portion and the second camera image portion.

[0093] The camera processor 14 can control the camera to focus the first camera image portion and the second camera image portion based on applying primary PDAF to the first camera image portion and not applying secondary PDAF to the first camera image portion and based on applying primary PDAF and secondary PDAF to the second camera image portion (108). For example, the camera processor 14 can control the positioning of the positioning of the lens 34 relative to the sensor 36 such that light rays refracted by the lens 34 intersect the sensor 36 at the appropriate locations. In some examples, it can be beneficial for the camera processor 14 to focus the second camera image portion sufficiently using secondary PDAF. In some examples, it can be beneficial for the camera processor 14 to process the first camera image portion using primary PDAF without using secondary PDAF because primary PDAF is sufficient to focus the first camera image portion without using secondary PDAF.

[0094] Additional aspects of the disclosure are detailed in the numbered clauses below.

[0095] Clause 1 - In one example, a system includes one or more memories configured to store a camera image captured with a camera and processing circuitry in communication with the one or more memories. The processing circuitry is configured to identify a first camera image portion and a second camera image portion in the camera image, apply primary PDAF to the first camera image portion, and apply the primary PDAF and secondary PDAF to the second camera image portion. Additionally, the processing circuitry is configured to control the camera to focus the first camera image portion and the second camera image portion based on applying primary PDAF to the first camera image portion and not applying secondary PDAF to the first camera image portion and based on applying primary PDAF and secondary PDAF to the second camera image portion.

[0096] Clause 2 - The system of clause 1, wherein the camera image includes a set of image regions, and wherein to identify the first camera image portion and the second camera image portion, the processing circuitry is configured to determine a set of confidence scores, wherein each confidence score of the set of confidence scores corresponds to an image region of the set of image regions of the camera image, identify the first camera image portion to include each image region of the set of image regions that corresponds to a confidence score that is greater than a threshold confidence score, and identify the second camera image portion to include each image region of the set of image regions that corresponds to a confidence score that is not greater than the threshold confidence score.

[0097] Clause 3 - The system of clause 2, wherein to determine the set of confidence scores, the processing circuitry is configured to: for each image region of the set of image regions corresponding to the camera image, identify one or more pixel values of a SAD curve corresponding to each pixel of the camera image, wherein the one or more pixel values include one or both of a color value and an intensity value; and determine each confidence score of the set of confidence scores based on the SAD curve corresponding to the respective image region.

[0098] Clause 4 - The system of clause 3, wherein to determine each confidence score of the set of confidence scores based on the SAD curve corresponding to the respective image region, the processing circuitry is configured to: identify a minimum value of the SAD curve corresponding to each image region of the set of image regions; calculate a mean value of the SAD curve corresponding to each image region of the set of image regions; and calculate each confidence score of the set of confidence scores based on the minimum value of the SAD curve corresponding to the respective image region and based on the mean value of the SAD curve corresponding to the respective image region.

[0099] Clause 5 - The system of any one of clauses 1-5, wherein the primary PDAF comprises a horizontal PDAF, and wherein the secondary PDAF comprises a vertical PDAF.

[0100] Clause 6 - The system of clause 5, wherein the system further comprises a hardware unit, and wherein the processing circuitry is configured to: apply the horizontal PDAF to the first camera image portion using the hardware unit; and apply the horizontal PDAF to the second camera image portion using the hardware unit.

[0101] Clause 7 - The system of clause 6, wherein the one or more memories are further configured to store vertical PDAF software, and wherein the processing circuitry is further configured to execute the vertical PDAF software to apply the vertical PDAF to the second camera image portion.

[0102] Clause 8 - The system of clause 7, wherein the one or more memories are further configured to store a vertical PDAF DNN, and wherein the processing circuitry is further configured to execute the vertical PDAF DNN to apply the vertical PDAF to the second camera image portion.

[0103] Clause 9 - The system of any of clauses 6-9, wherein the hardware unit is a first hardware unit, wherein the system further comprises a second hardware unit, and wherein the processing circuitry is further configured to apply the vertical PDAF to the second camera image portion using the second hardware unit.

[0104] Clause 10 - The system of any of clauses 1-9, wherein the primary PDAF comprises a hardware PDAF, wherein the secondary PDAF comprises a software PDAF, wherein the camera image comprises a long-exposure component and a short-exposure component, wherein to identify the first camera image portion and the second camera image portion, the processing circuitry is configured to: determine whether a ROI of the long-exposure component satisfies one or more criteria; identify the first camera image portion to include the long-exposure component; and identify the second camera image portion to include the short-exposure component corresponding to a portion of the ROI of the long-exposure component that does not satisfy the one or more criteria.

[0105] Clause 11 - The system of clause 10, wherein to determine whether the ROI of the long-exposure component satisfies the one or more criteria, the processing circuitry is configured to: determine a confidence score of the ROI; and determine whether the confidence score of the ROI is greater than a threshold confidence score.

[0106] Clause 12 - The system of any of clauses 10-11, wherein to determine whether the ROI of the long-exposure component satisfies the one or more criteria, the processing circuitry is configured to: determine an exposure level of the ROI; and determine whether the exposure level of the ROI is greater than a threshold exposure level.

[0107] Clause 13 - The system of any of clauses 1-12, the system further comprising the camera, and wherein the processing circuitry is further configured to control the camera to capture the camera image.

[0108] Clause 14 - A method comprising: identifying a first camera image portion and a second camera image portion in a camera image, applying a PDAF to the first camera image portion, and applying a primary PDAF and a secondary PDAF to the second camera image portion. The method further comprises: controlling the camera to focus the first camera image portion and the second camera image portion based on applying the primary PDAF to the first camera image portion and not applying the secondary PDAF to the first camera image portion and based on applying the primary PDAF and the secondary PDAF to the second camera image portion.

[0109] Clause 15 - The method of clause 14, wherein the camera image comprises a set of image regions, and wherein identifying the first camera image portion and the second camera image portion comprises: determining a set of confidence scores, wherein each confidence score of the set of confidence scores corresponds to an image region of the set of image regions of the camera image; identifying the first camera image portion to include each image region of the set of image regions that corresponds to a confidence score that is greater than a threshold confidence score; and identifying the second camera image portion to include each image region of the set of image regions that corresponds to a confidence score that is not greater than the threshold confidence score.

[0110] Clause 16 - The method of clause 15, wherein determining the set of confidence scores comprises: for each image region of the set of image regions corresponding to the camera image, identifying one or more pixel values of a SAD curve corresponding to each pixel of the camera image, wherein the one or more pixel values comprise one or both of a color value and an intensity value; and determining each confidence score of the set of confidence scores based on the SAD curve corresponding to the respective image region.

[0111] Clause 17 - The method of clause 16, wherein determining each confidence score of the set of confidence scores based on the SAD curve corresponding to the respective image region comprises: identifying a minimum value of the SAD curve corresponding to each image region of the set of image regions; calculating an average value of the SAD curve corresponding to each image region of the set of image regions; and calculating each confidence score of the set of confidence scores based on the minimum value of the SAD curve corresponding to the respective image region and based on the average value of the SAD curve corresponding to the respective image region.

[0112] Clause 18 - The method of any one of clauses 14 to 18, wherein the primary PDAF comprises a horizontal PDAF, and wherein the secondary PDAF comprises a vertical PDAF.

[0113] Clause 19 - The method of clause 18, wherein the system further comprises a hardware unit, and wherein the method further comprises: using the hardware unit to apply the horizontal PDAF to the first camera image portion; and using the hardware unit to apply the horizontal PDAF to the second camera image portion.

[0114] Clause 20 - The method of clause 19, wherein the one or more memories are further configured to store vertical PDAF software, and wherein the method further comprises executing the vertical PDAF software to apply the vertical PDAF to the second camera image portion.

[0115] Clause 21 - The method of clause 20, wherein the one or more memories are further configured to store a vertical PDAF DNN, and wherein the method further comprises executing the vertical PDAF DNN to apply the vertical PDAF to the second camera image portion.

[0116] Clause 22 - The method of any of clauses 19 to 21, wherein the hardware unit is a first hardware image processing unit, wherein the system further comprises a second hardware unit, and wherein the processing circuitry is further configured to use the second hardware unit to apply the vertical PDAF to the second camera image portion.

[0117] Clause 23 - The method of any of clauses 14 to 21, wherein the primary PDAF comprises a hardware PDAF, wherein the secondary PDAF comprises a software PDAF, wherein the camera image comprises a long-exposure component and a short-exposure component, and wherein identifying the first camera image portion and the second camera image portion comprises: determining whether a ROI of the long-exposure component satisfies one or more criteria; identifying the first camera image portion to include the long-exposure component; and identifying the second camera image portion to include the short-exposure component corresponding to a portion of the ROI of the long-exposure component that does not satisfy the one or more criteria.

[0118] Clause 24 - The method of clause 23, wherein determining whether the ROI of the long-exposure component satisfies the one or more criteria comprises: determining a confidence score of the ROI; and determining whether the confidence score of the ROI is greater than a threshold confidence score.

[0119] Clause 25 - The method of any of clauses 23 to 24, wherein determining whether the ROI of the long-exposure component satisfies the one or more criteria comprises: determining an exposure level of the ROI; and determining whether the exposure level of the ROI is greater than a threshold exposure level.

[0120] Clause 26 - The method of any of claims 14 to 24, the method further comprising the camera, and wherein the method further comprises controlling the camera to capture the camera image.

[0121] Clause 27 - A computer-readable medium comprising instructions that, when applied by processing circuitry, cause the processing circuitry to: identify a first camera image portion and a second camera image portion in a camera image, apply PDAF to the first camera image portion, and apply the primary PDAF and a secondary PDAF to the second camera image portion; and the instructions further cause the processing circuitry to: control the camera to focus the first camera image portion and the second camera image portion based on applying the primary PDAF to the first camera image portion and not applying the secondary PDAF to the first camera image portion and based on applying the primary PDAF and the secondary PDAF to the second camera image portion.

[0122] It is recognized that, in accordance with examples, certain acts or events of any of the techniques described herein can be performed in a different sequence, can be added, merged, or entirely omitted (e.g., all described acts or events can not be necessary to carry out the techniques). Moreover, in certain examples, acts or events can be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.

[0123] In one or more examples, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer- readable media generally can correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this disclosure. A computer program product can include a computer-readable medium.

[0124] By way of example, and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any

[0125] Instructions can be applied by one or more processors, such as one or more DSPs, general purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuitry. Accordingly, the terms "processor" and "processing circuitry," as used herein can refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0126] The techniques of this disclosure can be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require

[0127] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A system comprising: One or more memories configured to store camera images captured by the camera; and Processing circuitry, which communicates with the one or more memories, wherein the processing circuitry is configured to: The first camera image portion and the second camera image portion are identified in the camera images; Apply principal phase detection autofocus (PDAF) to the first camera image portion; The primary PDAF and secondary PDAF are applied to the image portion of the second camera; as well as The camera is controlled to focus on the first camera image portion and the second camera image portion by applying the primary PDAF to the first camera image portion and not applying the secondary PDAF to the first camera image portion, and by applying the primary PDAF and the secondary PDAF to the second camera image portion.

2. The system of claim 1, wherein the camera image comprises a set of image regions, and wherein, in order to identify the first camera image portion and the second camera image portion, the processing circuit is configured to: A set of confidence scores is determined, wherein each confidence score in the set of confidence scores corresponds to an image region in the set of image regions of the camera image; Identify the first camera image portion by each image region in the set of image regions corresponding to a confidence score greater than a threshold confidence score; as well as The second camera image portion is identified by each image region in the set of image regions corresponding to a confidence score that is not greater than the threshold confidence score.

3. The system of claim 2, wherein, in order to determine the set of confidence scores, the processing circuitry is configured to: For each image region in the set of image regions corresponding to the camera image, identify one or more pixel values ​​based on a sum of absolute differences (SAD) curve corresponding to each pixel of the camera image, wherein the one or more pixel values ​​include one or both of color values ​​and intensity values; and Each confidence score in the set is determined based on the SAD curve corresponding to the corresponding image region.

4. The system of claim 3, wherein, in order to determine each confidence score in the set of confidence scores based on the SAD curve corresponding to the corresponding image region, the processing circuit is configured to: The minimum value of the SAD curve corresponding to each image region in the set of image regions; Calculate the average value of the SAD curve corresponding to each image region in the set of image regions; and Each confidence score in the set is calculated based on the minimum value of the SAD curve corresponding to the corresponding image region and based on the average value of the SAD curve corresponding to the corresponding image region.

5. The system of claim 1, wherein the primary PDAF comprises a horizontal PDAF, and wherein the secondary PDAF comprises a vertical PDAF.

6. The system of claim 5, wherein the system further comprises a hardware unit, and wherein the processing circuit is configured to: The hardware unit is used to apply the horizontal PDAF to the first camera image portion; and The hardware unit is used to apply the horizontal PDAF to the second camera image portion.

7. The system of claim 6, wherein the one or more memories are further configured to store vertical PDAF software, and wherein the processing circuitry is further configured to execute the vertical PDAF software to apply the vertical PDAF to the second camera image portion.

8. The system of claim 7, wherein the one or more memories are further configured to store a vertical PDAF deep neural network (DNN), and wherein the processing circuitry is further configured to execute the vertical PDAF DNN to apply the vertical PDAF to a portion of the second camera image.

9. The system of claim 6, wherein the hardware unit is a first hardware unit, wherein the system further comprises a second hardware unit, and wherein the processing circuitry is further configured to use the second hardware unit to apply the vertical PDAF to the second camera image portion.

10. The system of claim 1, wherein the primary PDAF comprises hardware PDAF, wherein the secondary PDAF comprises software PDAF, wherein the camera image comprises a long exposure component and a short exposure component, wherein, in order to identify the first camera image portion and the second camera image portion, the processing circuit is configured to: Determine whether the long exposure component, including the region of interest (ROI), satisfies one or more criteria; Identify the first camera image portion to include the long exposure component; as well as The second camera image portion is identified to include the short exposure component corresponding to the portion of the ROI of the long exposure component that does not satisfy one or more of the criteria.

11. The system of claim 10, wherein, in order to determine whether the long exposure component including the ROI satisfies one or more criteria, the processing circuitry is configured to: Determine the confidence score of the ROI; and Determine whether the confidence score of the ROI is greater than the threshold confidence score.

12. The system of claim 10, wherein, in order to determine whether the long exposure component including the ROI satisfies one or more of the criteria, the processing circuitry is configured to: Determine the exposure level of the ROI; and Determine whether the exposure level of the ROI is greater than the threshold exposure level.

13. The system of claim 1, further comprising the camera, wherein the processing circuitry is further configured to control the camera to capture camera images.

14. A method, the method comprising: Identify the first camera image portion and the second camera image portion in the camera images; Apply principal phase detection autofocus (PDAF) to the first camera image portion; The primary PDAF and secondary PDAF are applied to the image portion of the second camera; as well as The camera is controlled to focus on the first camera image portion and the second camera image portion by applying the primary PDAF to the first camera image portion and not applying the secondary PDAF to the first camera image portion, and by applying the primary PDAF and the secondary PDAF to the second camera image portion.

15. The method of claim 14, wherein the camera image comprises a set of image regions, and wherein identifying the first camera image portion and the second camera image portion comprises: A set of confidence scores is determined, wherein each confidence score in the set of confidence scores corresponds to an image region in the set of image regions of the camera image; Identify the first camera image portion by each image region in the set of image regions corresponding to a confidence score greater than a threshold confidence score; as well as The second camera image portion is identified by each image region in the set of image regions corresponding to a confidence score that is not greater than the threshold confidence score.

16. The method of claim 15, wherein determining the set of confidence scores comprises: For each image region in the set of image regions corresponding to the camera image, identify one or more pixel values ​​based on a sum of absolute differences (SAD) curve corresponding to each pixel of the camera image, wherein the one or more pixel values ​​include one or both of color values ​​and intensity values; and Each confidence score in the set is determined based on the SAD curve corresponding to the corresponding image region.

17. The method of claim 16, wherein determining each confidence score in the set based on the SAD curve corresponding to the respective image region comprises: The minimum value of the SAD curve corresponding to each image region in the set of image regions; Calculate the average value of the SAD curve corresponding to each image region in the set of image regions; as well as Each confidence score in the set is calculated based on the minimum value of the SAD curve corresponding to the corresponding image region and based on the average value of the SAD curve corresponding to the corresponding image region.

18. The method of claim 14, wherein the primary PDAF comprises a horizontal PDAF, and wherein the secondary PDAF comprises a vertical PDAF.

19. The method of claim 18, wherein the system further comprises a hardware unit, and wherein the method further comprises: The hardware unit is used to apply the horizontal PDAF to the first camera image portion; as well as The hardware unit is used to apply the horizontal PDAF to the second camera image portion.

20. The method of claim 19, wherein the one or more memories are further configured to store vertical PDAF software, and wherein the method further comprises executing the vertical PDAF software to apply the vertical PDAF to a portion of the second camera image.

21. The method of claim 20, wherein the one or more memories are further configured to store a vertical PDAF deep neural network (DNN), and wherein the method further comprises executing the vertical PDAF DNN to apply the vertical PDAF to a portion of the second camera image.

22. The method of claim 19, wherein the hardware unit is a first hardware image processing unit, wherein the system further comprises a second hardware unit, and wherein the processing circuitry is further configured to use the second hardware unit to apply the vertical PDAF to a portion of the second camera image.

23. The method of claim 14, wherein the primary PDAF comprises hardware PDAF, wherein the secondary PDAF comprises software PDAF, wherein the camera image comprises a long exposure component and a short exposure component, and wherein identifying the first camera image portion and the second camera image portion comprises: Determine whether the long exposure component, including the region of interest (ROI), meets one or more criteria; Identify the first camera image portion to include the long exposure component; as well as The second camera image portion is identified to include the short exposure component corresponding to the portion of the ROI of the long exposure component that does not satisfy one or more of the criteria.

24. The method of claim 23, wherein determining whether the long exposure component including the ROI satisfies one or more of the criteria comprises: Determine the confidence score of the ROI; as well as Determine whether the confidence score of the ROI is greater than the threshold confidence score.

25. The method of claim 23, wherein determining whether the long exposure component including the ROI satisfies one or more of the criteria comprises: Determine the exposure level of the ROI; as well as Determine whether the exposure level of the ROI is greater than the threshold exposure level.

26. The method of claim 14, further comprising the camera, and wherein the method further comprises controlling the camera to capture camera images.

27. A computer-readable medium storing instructions that, when applied by processing circuitry, cause the processing circuitry to: Identify the first camera image portion and the second camera image portion in the camera images; Apply principal phase detection autofocus (PDAF) to the first camera image portion; The primary PDAF and secondary PDAF are applied to the image portion of the second camera; as well as The camera is controlled to focus on the first camera image portion and the second camera image portion by applying the primary PDAF to the first camera image portion and not applying the secondary PDAF to the first camera image portion, and by applying the primary PDAF and the secondary PDAF to the second camera image portion.

Images