Systems and methods for camera zoom - Patents.com

Abstract

Examples of digital zoom that preserve image characteristics, such as sharpness, clarity, and / or contrast, are described. In some aspects, a device can receive an image and determine various image characteristic scores corresponding to digitally zoomed versions of the image having different zoom intensities. For example, the device can determine a first image characteristic score for a first zoom intensity and a second image characteristic score for a second zoom intensity. The device can compare the image characteristic scores to an image characteristic threshold and can select a highest zoom intensity for which the corresponding image characteristic score does not fall below the image characteristic threshold. For example, the device can select a first zoom intensity if the first image characteristic score meets or exceeds the image characteristic threshold and the second image characteristic score does not meet or exceed the image characteristic threshold. The device can output image data corresponding to the image at the selected zoom intensity.

Description

[Technical field]

[0001]

[0001] The present application relates to image processing. More particularly, aspects of the present application relate to systems and methods for identifying high strength or levels of digital zoom for an image, where the image retains at least a threshold level of one or more image characteristics, such as sharpness, contrast, or clarity. [Background technology]

[0002] In optical systems, zooming refers to a technique that allows a camera to capture an image that appears to have been captured closer to or farther from the photographed object, depending on the zoom setting. Zooming in refers to adjusting the captured image to appear to have been captured closer to the photographed object. Zooming out refers to adjusting the captured image to appear to have been captured farther from the photographed object. Zooming includes optical zoom and digital zoom. Optical zoom produces clear images, but involves physical movement of one or more lenses of the camera. Digital zoom traditionally involves cropping and upscaling portions of the captured image. Digital zoom does not require specialized camera equipment. However, digitally zooming an image at high zoom strengths may adversely affect image characteristics such as sharpness, contrast, and / or clarity for the digitally zoomed portion of the image compared to the corresponding image characteristics of the original image. Summary of the Invention

[0003] Systems and techniques for digital zoom are described herein. In some aspects, a device may receive an image captured by an image sensor. The device may determine various image characteristic scores corresponding to digitally zoomed versions of the image having different zoom intensities (also referred to as zoom levels). For example, the device may determine a first image characteristic score for a first zoom intensity (or zoom level) and a second image characteristic score for a second zoom intensity (or zoom level). The device may compare the image characteristic scores to an image characteristic threshold and may select a highest zoom intensity for which the corresponding image characteristic score is not below the image characteristic threshold. In some examples, the device may select a highest zoom intensity for which the corresponding image characteristic score is above the image characteristic threshold. In some examples, the device may select a highest zoom intensity for which the corresponding image characteristic score is equal to the image characteristic threshold. For example, the device may select a first zoom intensity if the first image characteristic score meets or exceeds the image characteristic threshold and the second image characteristic score is below the image characteristic threshold. The device can output image data corresponding to the digitally zoomed portion of the image at a selected zoom intensity (e.g., display the image data, store the image data, send the image data to another device, etc.).

[0004] In one example, an apparatus for image processing is provided. The apparatus includes a memory and one or more processors coupled to the memory (e.g., implemented in a circuit). The one or more processors are configured and capable of receiving an image from an image sensor; determining a first image characteristic score for a first set of image data, the first set of image data corresponding to a first zoomed variant of the image at a first zoom intensity; determining a second image characteristic score for a second set of image data, the second set of image data corresponding to a second zoomed variant of the image at a second zoom intensity; identifying that the second image characteristic score is less than an image characteristic threshold; and outputting the first set of image data as an output image based on identifying that the second image characteristic score is less than the image characteristic threshold.

[0005] In another example, a method of image processing is provided. The method includes receiving an image captured by an image sensor. The method includes receiving the image from the image sensor; determining a first image characteristic score for a first set of image data, the first set of image data corresponding to a first zoomed variant of the image at a first zoom intensity; determining a second image characteristic score for a second set of image data, the second set of image data corresponding to a second zoomed variant of the image at a second zoom intensity; identifying that the second image characteristic score is less than an image characteristic threshold; and outputting the first set of image data as an output image based on identifying that the second image characteristic score is less than the image characteristic threshold.

[0006]

[0006] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon which, when executed by one or more processors, cause the one or more processors to: receive an image from an image sensor; determine a first image characteristic score for a first set of image data, the first set of image data corresponding to a first zoomed variant of the image at a first zoom intensity; determine a second image characteristic score for a second set of image data, the second set of image data corresponding to a second zoomed variant of the image at a second zoom intensity; identify that the second image characteristic score is less than an image characteristic threshold; and output the first set of image data as an output image based on identifying that the second image characteristic score is less than the image characteristic threshold.

[0007] In another example, an apparatus for image processing is provided that includes means for receiving an image from an image sensor; means for determining a first image characteristic score of a first set of image data, the first set of image data corresponding to a first zoomed variant of the image at a first zoom intensity; means for determining a second image characteristic score of a second set of image data, the second set of image data corresponding to a second zoomed variant of the image at a second zoom intensity; means for identifying that the second image characteristic score is less than an image characteristic threshold; and means for outputting the first set of image data as an output image based on identifying that the second image characteristic score is less than the image characteristic threshold.

[0008] In some aspects, outputting the first set of image data as the output image is also based on the first image characteristic score being greater than or equal to an image characteristic threshold value.

[0009] In some aspects, the first image characteristic score is a first image sharpness score, the second image characteristic score is a second image sharpness score, and the image characteristic threshold is an image sharpness threshold. In some aspects, the first image characteristic score is a first under-sharpening value of the first set of image data, the second image characteristic score is a second under-sharpening value of the second set of image data, and the image characteristic threshold is an under-sharpening value threshold.

[0010] In some aspects, the methods, apparatus, and computer-readable media described above further comprise receiving one or more inputs identifying a portion of the image, where the first zoomed version of the image and the second zoomed version of the image are based on the identified portion of the image. In some aspects, the first zoomed version of the image and the second zoomed version of the image include the identified portion of the image. In some aspects, the one or more inputs include a user input through a user interface. In some aspects, the one or more inputs include at least one of a touch input, a hover input, a gesture input, and a gaze input. In some aspects, the methods, apparatus, and computer-readable media described above further comprise receiving one or more inputs from an object detection algorithm, where the object detection algorithm determines that the identified portion of the image includes a depiction of a type of object. In some aspects, the type of object is a face.

[0011]

[0011] In some aspects, the methods, apparatus, and computer-readable media described above further comprise generating a first zoomed version of the image, at least in part, by cropping and upscaling the image according to a first zoom intensity, and generating a second zoomed version of the image, at least in part, by cropping and upscaling at least one of the image and the first zoomed version of the image according to a second zoom intensity.

[0012] In some aspects, to output the first set of image data as an output image, the one or more processors are configured to render the output image for display on a display screen. In some aspects, the apparatus described above further comprises a display screen configured to display the output image.

[0013]

[0013] In some aspects, the methods, apparatus, and computer-readable media described above further include identifying that a difference between the first image characteristic score and the second image characteristic score is less than a difference threshold, and outputting a second set of image data as a second output image based on the difference being less than the difference threshold.

[0014] In some aspects, the first set of image data corresponds to a first digitally zoomed portion of the image at a first zoom intensity and the second set of image data corresponds to a second digitally zoomed portion of the image at a second zoom intensity. In some aspects, the second digitally zoomed portion of the image is a subset of the first digitally zoomed portion of the image. In some aspects, the first digitally zoomed portion of the image is different from the second digitally zoomed portion of the image.

[0015] In some aspects, to output the first set of image data as the output image, the one or more processors are configured to transmit the output image using the communication interface. In some aspects, the apparatus described above further comprises a communication interface configured to transmit the output image to a device.

[0016] In some aspects, the apparatus described above is one of a mobile device, a mobile handset, a wireless communication device, a head mounted display, and a camera. In some aspects, the apparatus described above further comprises an image sensor configured to capture an image.

[0017] In some aspects, the apparatus comprises a camera, a mobile device, a mobile phone, a smartphone, a mobile handset, a portable gaming device, a wireless communication device, a smart watch, a wearable device, a head mounted display (HMD), an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, the one or more processors include an image signal processor (ISP). In some aspects, the apparatus includes one or more cameras for capturing one or more images. In some aspects, the apparatus includes an image sensor for capturing the images. In some aspects, the apparatus further includes a display for displaying the image, one or more notifications (e.g., associated with processing the image), and / or other displayable data. In some aspects, the display displays the image after the one or more processors have processed the image.

[0018]

[0018] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0019]

[0019] The foregoing, together with other features and embodiments, will become more apparent from the following specification, claims, and accompanying drawings.

[0020]

[0020] Exemplary embodiments of the present application are described in detail below with reference to the following figures. [Brief description of the drawings]

[0021] [Figure 1]

[0021] FIG. 1 is a block diagram showing the architecture of an image capture and processing system. [Diagram 2]

[0022] FIG. 1 is a block diagram illustrating the architecture of an image processing system including an image characterization engine. [Figure 3A]

[0023] 1 is a conceptual diagram illustrating a selection interface in which a digitally zoomed portion of an image is centered on a portion of the image identified based on touch input. [Figure 3B]

[0024] 1 is a conceptual diagram illustrating a selection interface in which a digitally zoomed portion of an image is centered on a portion of the image that has been identified based on image analysis. [Figure 4A]

[0025] 1 is a conceptual diagram showing an image and a corresponding sharpness graph. [Figure 4B]

[0026] FIG. 4B is a conceptual diagram showing a first digitally zoomed portion of the image of FIG. 4A and a corresponding sharpness graph. [Figure 4C]

[0027] FIG. 4B is a conceptual diagram showing a second digitally zoomed portion of the image of FIG. 4A and a corresponding sharpness graph. [Diagram 5]

[0028] 1 is a conceptual diagram illustrating different zoom strengths and corresponding feature scores along a series of incremental zoom movements. [Figure 6]

[0029] 4 is a flow diagram illustrating operations for processing image data. [Figure 7]

[0030] FIG. 1 illustrates an example of a system for implementing some aspects of the present technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022]

[0031] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments can be applied independently, and some of them can be applied in combination. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be implemented without these specific details. The figures and descriptions are not limiting.

[0023]

[0032] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application, as set forth in the appended claims.

[0024]

[0033] An image capture device (e.g., a camera) is a device that uses an image sensor to receive light and capture an image frame, such as a still image or a video frame. The terms "image," "image frame," and "frame" are used interchangeably herein. An image capture device typically includes at least one lens that receives light from a scene and bends the light toward an image sensor of the image capture device. The light received by the lens passes through an aperture controlled by one or more control mechanisms and is received by the image sensor. The one or more control mechanisms can control exposure, focus, and / or zoom based on information from the image sensor and / or based on information from an image processor (e.g., a host or application process and / or an image signal processor). In some examples, the one or more control mechanisms include a motor or other control mechanism that moves a lens of the image capture device to a target lens position.

[0025]

[0034] Zoom refers to a set of techniques that allow a camera to capture an image that appears to have been captured from closer to the photographed object or farther away from the photographed object, depending on the zoom setting. Zooming in refers to adjusting the captured image to make it appear to have been captured from closer to the photographed object. Zooming out refers to adjusting the captured image to make it appear to have been captured from farther away from the photographed object. Zoom includes optical zoom and digital zoom.

[0026]

[0035] Optical zoom is a technique in which zooming is accomplished by changing the focal length of a camera's zoom lens, or by switching from a camera with a first focal length to a camera with a second focal length. Zooming in through optical zoom does not degrade image quality, but requires special camera equipment and must be performed before the image is captured.

[0027]

[0036] Digital zoom is a technique for zooming in on a portion of an image by cropping the image and upscaling the cropped image. Digital zoom can be beneficial because it does not require specialized camera equipment such as a telephoto lens, and therefore can enable zoom functionality on devices with fairly basic cameras. Digital zoom can also be used in addition to optical zoom to zoom in beyond the optical zoom capabilities of a particular camera. Digital zoom can be performed at any point after an image is captured. However, zooming in using digital zoom involves cropping a portion of the original image and upscaling it. The process of zooming in can degrade image characteristics, such as sharpness, contrast, and / or clarity. For example, a cropped and upscaled portion of an image can have a lower image characteristic score (e.g., lower sharpness, lower contrast, and / or lower clarity) than the original image. For example, the quality degradation introduced by digital zoom often reduces the sharpness of edges (e.g., making edges appear blurry) and / or reduces the smoothness of curves (e.g., making curves appear to have jagged edges).

[0028]

[0037] As described in more detail below, systems and techniques are described herein for determining a selected digital zoom level that preserves at least a threshold level of one or more image characteristics, such as sharpness, contrast, and / or clarity. For example, an image sensor (e.g., a camera or portion of a camera) of a device may capture an image. An input identifying a portion of the image may be received at the device. For example, the device may receive the input while a display screen of the device displays an image. In some examples, the input may be a touch input, for example, touching a portion of the image displayed on the display screen if the display screen is a touch screen. The input may be a hover input, an eye-tracking input, a mouse cursor click input, a pointer click input, a button press input, any combination thereof, and / or another type of input that identifies or selects a portion of the image.

[0029]

[0038] A device can generate multiple zoomed images from an image using digital zoom at different zoom strengths (also referred to as zoom levels), such that the zoomed images include portions of the image. For example, the device can generate a first zoomed image by digitally zooming in on a portion of the image using a first zoom strength (e.g., 2X zoom). The device can generate a second zoomed image by digitally zooming in on a portion of the image using a second zoom strength (e.g., 4X zoom) that is greater than the first zoom strength. Thus, the second zoomed image is cropped and upscaled at a higher zoom strength than the first zoomed image.

[0030]

[0039] The device can determine an image sharpness value for each of the zoomed images. For example, the device can determine a first image sharpness for a first zoomed image and a second image sharpness for a second zoomed image. Each of the image sharpness values ​​can be, for example, an undersharpening percentage calculated by the device using a modulo transfer function (MTF). The image sharpness measurements can be based on other measurements, such as line pairs per millimeter (mm) measurements.

[0031]

[0040] The predetermined image characteristic threshold value may be stored in the device. The device may compare different image characteristic scores to the image characteristic threshold value. The zoomed image with the greatest zoom strength whose image characteristic score value is still greater than or equal to the image characteristic threshold value is selected as the selected zoomed image. The image characteristic score may be an image sharpness score, and the image characteristic threshold value may be an image sharpness threshold value. In one illustrative example, the image sharpness threshold value may be 80% under-sharpening. In this example, if a first zoomed image with 1.5x zoom has a first image sharpness score of 95% undersharpening, a second zoomed image with 2x zoom has an image sharpness score of 86% undersharpening, and a third zoomed image with 2.5x zoom has an image sharpness score of 72% undersharpening, then the second zoomed image (at 2x zoom) is the selected zoomed image because the 86% image sharpness score for the second zoomed image is greater than the 80% sharpness threshold and the 72% image sharpness score for the third zoomed image is less than the 80% sharpness threshold.

[0032]

[0041] The systems and techniques described herein provide technical improvements to imaging techniques that apply digital zoom techniques. For example, the systems and techniques described herein maximize digital zoom strength while retaining high levels of image characteristics such as sharpness, contrast, and clarity. This provides the benefits of digital zoom, such as the ability to perform zoom operations without specialized hardware (e.g., telephoto lenses) while retaining high image quality.

[0033]

[0042] 1 is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components used to capture and process an image of a scene (e.g., an image of a scene 110). The image capture and processing system 100 can capture a standalone image (or photograph) and / or can capture a video that includes multiple images (or video frames) in a particular sequence. A lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward an image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0034]

[0043] The one or more controls 120 may control exposure, focus, and / or zoom based on information from image sensor 130 and / or based on information from image processor 150. The one or more controls 120 may include multiple mechanisms and components, for example, the control 120 may include one or more exposure controls 125A, one or more focus controls 125B, and / or one or more zoom controls 125C. The one or more controls 120 may also include additional controls beyond those shown, such as controls to control analog gain, flash, HDR, depth of field, and / or other image capture properties.

[0035]

[0044] The focus control mechanism 125B of the control mechanism 120 can obtain the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can move the lens 115 closer to or farther from the image sensor 130 by actuating a motor or servo (or other lens mechanism), thereby adjusting the focus. In some cases, additional lenses, such as one or more microlenses on each photodiode of the image sensor 130, can be included in the system 100, each bending light received from the lens 115 toward a corresponding photodiode before the light reaches the photodiode. The focus setting can be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus settings may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus settings may be referred to as image capture settings and / or image processing settings.

[0036]

[0045] The exposure control 125A of the control mechanism 120 can obtain the exposure setting. In some cases, the exposure control 125A stores the exposure setting in a memory register. Based on this exposure setting, the exposure control 125A can control the size of the aperture (e.g., aperture size or f / stop), the duration the aperture is open (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0037]

[0046] The zoom control 125C of the control 120 can obtain the zoom setting. In some examples, the zoom control 125C stores the zoom setting in a memory register. In some examples, the zoom setting can include a digital zoom setting. In some examples, the zoom setting can include an optical zoom setting. Based on the zoom setting, the zoom control 125C can control which of the set of cameras is active, the focal length of an assembly of lens elements (lens assembly) including the lens 115 and / or one or more additional lenses, or a combination thereof. For example, the zoom control 125C can control the focal length of the lens assembly by actuating one or more motors or servos of the zoom control 125C) to move one or more of the lenses relative to one another. The zoom setting may be referred to as an image capture setting and / or an image processing setting.

[0038]

[0047] The image sensor 130 includes one or more arrays of photodiodes or other light-sensitive elements. Each photodiode measures an amount of light that ultimately corresponds to a particular pixel in the image produced by the image sensor 130. In some cases, different photodiodes may be covered by different color filters and may therefore measure light that matches the color of the filter covering the photodiode. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, and each pixel of the image is generated based on red light data from at least one photodiode covered with a red filter, blue light data from at least one photodiode covered with a blue filter, and green light data from at least one photodiode covered with a green filter. Other types of color filters may use yellow, magenta, and / or cyan (also called "emerald") color filters instead of or in addition to red, blue, and / or green color filters. Some image sensors (e.g., the image sensor 130) may not have color filters at all and may instead use different photodiodes across the entire pixel array (which in some cases are vertically stacked). Different photodiodes across the pixel array have different spectral sensitivity curves and can therefore respond to different wavelengths of light. Monochrome image sensors also lack color filters and therefore may lack color depth.

[0039]

[0048] In some cases, the image sensor 130 may alternatively or additionally include an opaque mask and / or a reflective mask that blocks light from reaching certain photodiodes, or portions of certain photodiodes, at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). The image sensor 130 may also include an analog gain amplifier for amplifying an analog signal output by the photodiode, and / or an analog-to-digital converter (ADC) for converting the analog signal output of the photodiode (and / or amplified by the analog gain amplifier) ​​to a digital signal. In some cases, some components or functions described with respect to one or more of the control mechanisms 120 may instead or in addition be included in the image sensor 130. The image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide semiconductor (CMOS), an N-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0040]

[0049] Image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISPs 154), one or more host processors (including host processor 152), and / or one or more of any other types of processors 710 described with respect to computing system 700. Host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-chip or SoC) that includes host processor 152 and ISPs 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) ports 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O ports 156 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter Integrated Circuit 2 (I2C) interface, an Inter Integrated Circuit 3 (I3C) interface, a serial peripheral interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface), an Advanced High-performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0041]

[0050] Image processor 150 may perform several tasks, such as de-mosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, auto white balance, merging image frames to form HDR images, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. Image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 / 720, read only memory (ROM) 145 / 725, a cache, a memory unit, another storage device, or some combination thereof.

[0042]

[0051] Various input / output (I / O) devices 160 may be connected to image processor 150. I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a track pad, a touch-sensitive surface, a printer, any other output device 735, any other input device 745, or some combination thereof. In some cases, captions may be entered into image processing device 105B through a physical keyboard or keypad of I / O device 160 or through a virtual keyboard or keypad of a touch screen of I / O device 160. I / O 160 may include one or more ports, jacks, or other connectors that enable a wired connection between system 100 and one or more peripheral devices, through which system 100 may receive data from and / or transmit data to one or more peripheral devices. I / O 160 may include one or more wireless transceivers that enable a wireless connection between system 100 and one or more peripheral devices, through which system 100 may receive data from and / or transmit data to one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 160 previously described, and may themselves be considered I / O devices 160 when coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0043]

[0052] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing system coupled to a camera). In some implementations, the image capture device 105A and the image processing device 105B may be coupled to each other, for example, via one or more wires, cables, or other electrical connectors and / or wirelessly via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B may be separate from each other.

[0044]

[0053] 1, a vertical dashed line divides the image capture and processing system 100 of FIG. 1 into two portions, which respectively represent an image capture device 105A and an image processing device 105B. Image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. Image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), a RAM 140, a ROM 145, and an I / O 160. In some cases, some components shown in image capture device 105A, such as ISP 154 and / or host processor 152, may be included in image capture device 105A.

[0045]

[0054] The image capture and processing system 100 may include an electronic device, such as a mobile or landline telephone handset (e.g., a smartphone, a cellular telephone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 wi-fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing system, such as a mobile handset, a desktop computer, or another computing system.

[0046]

[0055] Although image capture and processing system 100 is shown to include several components, one skilled in the art will appreciate that image capture and processing system 100 can include many more components than those shown in FIG. 1. The components of image capture and processing system 100 can include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of image capture and processing system 100 can include and / or be implemented using electronic circuitry or other electronic hardware that can include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits) and / or can include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein. The software and / or firmware can include one or more instructions stored in a computer-readable storage medium and executable by one or more processors of an electronic device implementing image capture and processing system 100.

[0047]

[0056] In some examples, optical zoom may be controlled using lens 115, zoom control 125C, image processor 150, ISP 154, and / or host processor 152. In some examples, digital zoom may be controlled using zoom control 125C, image processor 150, ISP 154, and / or host processor 152.

[0048]

[0057] FIG. 2 is a block diagram illustrating an architecture of an image processing system 200 including an image characterization engine 235. The image processing system 200 includes an image sensor 205. The image sensor 205 may be an example of an image sensor 130. In some examples, the image sensor 205 may include one or more additional components (e.g., the control mechanism 120). The image sensor 205 may capture image data corresponding to an image. The image may be sent from the image sensor 205 to an image signal processor (ISP) 210, which may receive the image from the image sensor 205. The ISP 210 may be an example of an ISP 154. In some examples, the ISP 210 may include one or more additional components (e.g., the image processor 150, the host processor 152, the I / O 156, the I / O 160, the ROM 145, the RAM 140). The ISP 210 may perform a number of processing operations on the image to at least partially process the image. For example, the ISP 210 may perform demosaicking color space conversion (e.g., from RGB to YUV), white balancing, black balancing, or a combination thereof. The ISP 210 may send images to a post-processing engine 215, which may receive images from the ISP 210. The ISP 210 may send images to an image characterization engine 235, which may receive images from the ISP 210.

[0049]

[0058] The post-processing engine 215 may be an example of the host processor 152. The post-processing engine 215 may be an example of the image processor 150. In some examples, the post-processing engine 215 may include one or more additional components (e.g., I / O 156, I / O 160, ROM 145, RAM 140). The post-processing engine 215 may further process the image. For example, the post-processing engine 215 may perform a cropping operation, a resizing operation, a digital zoom operation, or a combination thereof. The resizing operation may include, for example, an upscaling operation, an upsampling operation, a downscaling operation, a downsampling operation, a subsampling operation, a rescaling operation, a resampling operation, or a combination thereof. The resizing operation may include nearest neighbor (NN) rescaling, bilinear interpolation, bicubic interpolation, Sinc resampling, Lanczos resampling, box sampling, mipmapping, Fourier transform scaling, edge-directed interpolation, high quality scaling (HQX), a specialized context-sensitive rescaling technique, or a combination thereof. The post-processing engine 215 may send the image to the camera preview engine 220, which may receive the image from the post-processing engine 215. The post-processing engine 215 may send the image to the image characterization engine 235, which may receive the image from the post-processing engine 215. The post-processing engine 215 may send the image to the selection interface 230, which may receive the image from the post-processing engine 215. The post-processing engine 215 can send images to the media encoder 240, and the media encoder 240 can receive images from the post-processing engine 215.

[0050]

[0059] The image characterization engine 235 of the image processing system 200 can receive images from the ISP 210, from the post-processing engine 215, from the selection interface 230, from the camera preview engine 220, or from a combination thereof. Images received by the image characterization engine 235 can be processed by the ISP 210. Images received by the image characterization engine 235 can be cropped, resized, and / or otherwise processed by the post-processing engine 215. The image characterization engine 235 can generate an image characterization score for the image. The image characterization score can be a score indicative of the sharpness, contrast, and / or clarity of the image. In some examples, the characterization score can be an undersharpening percentage or an undersharpening value. In some examples, the image characterization engine 235 can generate a characterization score for the image using a modulo transfer function (MTF). In some examples, the characterization score can be a line pairs (lp) per millimeter (mm) measurement (lp / mm) value. In some examples, the image characterization engine 235 may generate a characterization score for the image based on an analysis of regions of interest in the image.

[0051]

[0060] The image characteristic evaluation engine 235 can send the image to the post-processing engine 215, which can receive the image from the image characteristic evaluation engine 235. The post-processing engine 215 can determine which zoom intensity should be used for the output image (e.g., to be encoded by the media encoder 240) based on whether the quality score for each zoom intensity applied to the image (as identified by the image characteristic evaluation engine 235) is less than, equal to, or greater than a predetermined quality threshold. The image characteristic evaluation engine 235 can send the image to the camera preview engine 220, which can receive the image from the image characteristic evaluation engine 235. The camera preview engine 220 can determine which zoom intensity should be used for the preview image based on whether the quality score for each zoom intensity applied to the image is less than, equal to, or greater than a predetermined quality threshold. The image characteristic evaluation engine 235 can send the image to the selection interface 230, which can receive the image from the image characteristic evaluation engine 235. The camera preview engine 220 can determine which zoom strength should be used by the selection interface 230 based on whether the quality score for each zoom strength applied to the image is less than, equal to, or greater than a predetermined quality threshold.

[0052]

[0061] The camera preview engine 220 of the image processing system 200 can receive images from the ISP 210, from the post-processing engine 215, from the selection interface 230, from the image characterization engine 235, or from a combination thereof. The camera preview engine 220 can generate a preview image based on the images it receives. The images received by the camera preview engine 220 can be processed by the ISP 210. The images received by the camera preview engine 220 can be cropped, resized, and / or otherwise processed by the post-processing engine 215. The images received by the camera preview engine 220 can be modified by the selection interface 230. The preview image can be displayed by the display screen 225 of the image processing system 200. The preview image can act as a viewfinder to preview the image data being captured by the image sensor 205. The preview image can be updated periodically and / or in real time. The camera preview engine 220 can send the preview image to the display screen 225, and the display screen 225 can receive the image from the camera preview engine 220. The camera preview engine 220 can send the preview image to the selection interface 230, which can receive the image from the camera preview engine 220. The camera preview engine 220 can send the preview image to the image characterization engine 235, which can receive the image from the camera preview engine 220.

[0053]

[0062] The selection interface 230 of the image processing system 200 can receive images from the ISP 210, from the post-processing engine 215, from the camera preview engine 220, from the image characterization engine 235, or from a combination thereof. The selection interface 230 can be associated with a touch-sensitive surface, such as the touch-sensitive surface of a touch screen. The display screen 225 can be a touch screen, or a portion of a touch screen. The selection interface 230 can be associated with a mouse cursor, a stylus, a keyboard, a keypad, a hover input detector, a gesture input detector, or another type of input device 745. The selection interface 230 can receive one or more inputs through an input interface. The input interface can correspond to the I / O 156 and / or the I / O 160. The selection interface 230 can identify a portion of the image based on one or more inputs (e.g., as in the case of the touch input 335 of FIG. 3A). The selection interface 230 may send the identified portions of the image and / or the one or more inputs themselves to the post-processing engine 215, which may receive the portions of the image from the selection interface 230. The post-processing engine 215 may crop, resize, and / or zoom the image based on the portions of the image identified using the one or more inputs.

[0054]

[0063] The selection interface 230 may be associated with a software application. For example, the selection interface 230 may be associated with a face detection algorithm, a face recognition algorithm, a face tracking algorithm, an object detection algorithm, an object recognition algorithm, an object tracking algorithm, a feature detection algorithm, a feature recognition algorithm, a feature tracking algorithm, or a combination thereof. The selection interface 230 may be associated with one or more artificial intelligence (AI) algorithms, one or more trained ML models based on one or more machine learning (ML) algorithms, one or more trained neural networks (NNs), or a combination thereof. The software application associated with the selection interface 230 may identify a portion of the image based on, for example, the software application detecting that the portion of the image includes a depiction of a particular type of feature, object, and / or face (e.g., as in FIG. 3B). The selection interface 230 may send the identified portion of the image to the post-processing engine 215, which may receive the portion of the image from the selection interface 230. The post-processing engine 215 may crop, resize, and / or zoom the image based on the portion of the image identified using the detected depiction.

[0055]

[0064] In some examples, the selection interface 230 of the image processing system 200 may also be used to modify an image. For example, the selection interface 230 may be associated with a software application through which a user may adjust processing settings for the image, annotate the image, manually instruct the post-processing engine 215 to crop the image, manually instruct the post-processing engine 215 to resize the image, manually instruct the post-processing engine 215 to zoom the image, rotate the image, make other changes to the image, or some combination thereof. The selection interface 230 may send the image and / or the input and / or the identified portion(s) of the image to the post-processing engine 215, which may receive the image from the selection interface 230. The selection interface 230 may send the image and / or the input and / or the identified portion(s) of the image to the image characterization engine 235, which may receive the image from the selection interface 230. The selection interface 230 can send the image and / or the input and / or the identified portion(s) of the image to the camera preview engine 220, and the camera preview engine 220 can receive the image from the selection interface 230.

[0056]

[0065] The media encoder 240 of the image processing system 200 can receive images from the ISP 210, from the post-processing engine 215, from the camera preview engine 220, from the image characterization engine 235, from the selection interface 230, or from a combination thereof. The images received by the camera preview engine 220 can be processed by the ISP 210. The images received by the camera preview engine 220 can be cropped, resized, and / or otherwise processed by the post-processing engine 215. The images received by the camera preview engine 220 can be modified by the selection interface 230. In some examples, the post-processing engine 215 can generate a zoomed-in version of the image at a zoom intensity that the post-processing engine 215 selects based on the characteristic scores for the different zoom intensities determined by the image characterization engine 235. The post-processing engine 215 can send the zoomed-in version of the image to the media encoder 240, and the media encoder 240 can receive the zoomed-in version of the image from the post-processing engine 215. The media encoder 240 may encode one or more images it receives using still image encoding techniques, such as still image compression techniques. The media encoder 240 may encode one or more images it receives using video encoding techniques, such as video compression techniques. The video encoding techniques may use the image as one of multiple video frames in a video and may involve inter-frame coding and / or intra-frame coding. The media encoder 240 may generate an encoded image. The encoded image may be referred to as an output image. In some cases, an image sent to the media encoder 240 (e.g., from the processing engine 215 and / or any other sending component described herein) may be referred to as an output image. The media encoder 240 may send images to the camera preview engine 220, and the camera preview engine 220 may receive images from the media encoder 240.The media encoder 240 can send images to the display screen 225, and the display screen 225 can receive images from the media encoder 240. The display screen 225 can display the images that it receives from the media encoder 240.

[0057]

[0066] The display screen 225 of the image processing system 200 may receive images from the media encoder 240, from the ISP 210, from the post-processing engine 215, from the camera preview engine 220, from the image characterization engine 235, from the selection interface 230, or from a combination thereof. The display screen 225 may display images. The display screen 225 may be any type of display screen or other display device (e.g., a projector). The display screen 225 may be or include elements described with respect to the output device(s) 735. For example, the display screen 225 may also include a speaker, which may output audio corresponding to the image (e.g., if the image is a video frame of a video that includes an audio track). The display screen 225 may correspond to the I / O 156 and / or the I / O 160.

[0058]

[0067] FIG. 3A is a conceptual diagram 300A illustrating a selection interface in which a digitally zoomed portion 320 of an image 310 is centered on an identified portion 330 of the image 310 based on a touch input 335. The image 310 illustrates an office environment in which four people are sitting or standing around a set of desks and chairs. A fingerprint icon is overlaid on the image 310 in FIG. 3A. The fingerprint icon represents a touch input 335 received through the selection interface (e.g., the selection interface 230). The selection interface may include a touch-sensitive surface, such as the touch-sensitive surface of a touch screen. The selection interface identifies the identified portion 330 of the image 310 that includes at least a subset of the area covered by the touch input 335 based on the location of the touch input 335 on the display screen and based on the positioning of the image 310 displayed on the display screen. The identified portion 330 of the image 310 is shown as a circular portion 330 of the image 310, outlined using a dashed line. In some cases, the identified portion 330 of the image 310 may simply be a point representing the center of the area covered by the touch input 335, or a point representing a cursor click, or a point representing a point touched by a stylus, or some combination thereof.

[0059]

[0068] The selection interface identifies the digitally zoomed portion 320 of the image 310 based on the identified portion 330 of the image 310. For example, the digitally zoomed portion 320 of the image 310 in FIG. 3A is centered on the identified portion 330 of the image 310. The digitally zoomed portion 320 of the image 310 is shown in FIG. 3A as a shaded rectangle outlined by a dashed line. The digitally zoomed portion 320 of the image 310 represents a crop area of ​​the image 310 for a digital zoom operation. For example, a digital zoom operation crops out everything outside the digitally zoomed portion 320 of the image 310, leaving only the digitally zoomed portion 320 of the image 310 remaining. The digital zoom operation can resize (e.g., upscale and / or upsample) the digitally zoomed portion 320 of the image 310 before or after cropping.

[0060]

[0069] FIG. 3B is a conceptual diagram 300B illustrating a selection interface in which a digitally zoomed portion 340 of the image 310 is centered on a portion 350 of the image 310 that was identified based on image analysis. The image 310 shown in FIG. 3B is the same image 310 of the office environment shown in FIG. 3A. The image analysis identifies a face of one of the people in the image 310 as represented by a dashed rounded rectangle shown around the face and identified as representing the identified portion 350 of the image 310. In some examples, the image analysis identifies the portion 350 of the image 310 by detecting the face using a face detection algorithm 355. In some examples, the image analysis identifies the portion 350 of the image 310 by recognizing the face using a face recognition algorithm 355. The identified portion 350 of the image 310 may be a bounding box around the face generated by the image analysis based on the detection of the face. The digitally zoomed portion 340 of the image 310 is based on the identified portion 350 of the image 310. For example, the digitally zoomed portion 340 of the image 310 in Figure 3B is centered on the identified portion 350 of the image 310. The digitally zoomed portion 340 of the image 310 represents a crop area of ​​the image 310 for the digital zoom operation. The digitally zoomed portion 340 of the image 310 is shown in Figure 3B as a shaded rectangle outlined by a dashed line.

[0061]

[0070] Although the digitally zoomed portion 340 of the image 310 is shown as being centered on the identified portion 350 of the image 310, that need not be the case. In some examples, the digitally zoomed portion 340 of the image 310 can include the identified portion 350 of the image 310 without being centered on the identified portion 350 of the image 310. In some examples, the digitally zoomed portion 340 of the image 310 can be identified based on the identified portion 350 of the image 310 without including the identified portion 350 of the image 310. For example, the identified portion 350 of the image 310 can represent a corner or a side of the digitally zoomed portion 340 of the image 310. Similarly, although the digitally zoomed portion 340 of the image 310 is shown as being centered on the identified portion 330 of the image 310, that need not be the case. In some examples, the digitally zoomed portion 340 of the image 310 may include the identified portion 330 of the image 310 without being centered on the identified portion 330 of the image 310. In some examples, the digitally zoomed portion 340 of the image 310 may be identified based on the identified portion 330 of the image 310 without including the identified portion 330 of the image 310. For example, the identified portion 330 of the image 310 may represent a corner or a side of the digitally zoomed portion 340 of the image 310.

[0062]

[0071] In some examples, a semantic analysis algorithm that performs a semantic analysis of the image 310 may be used instead of or in addition to the face recognition algorithm 355. The semantic analysis may, for example, perform a segmentation between one or more foreground elements and one or more background elements. The semantic analysis may perform the segmentation based on an analysis of the image itself, an analysis of other images of the same scene (e.g., previous image frames in a video of the scene), an analysis of depth information from a depth sensor (e.g., from a LIDAR, RADAR, SONAR, SODAR, time-of-flight sensor, structured light sensor, or combinations thereof), or a combination thereof. For example, the identified portion 350 of the image may be selected based on a semantic analysis of the image 310 indicating that the identified portion 350 of the image includes a depiction of a foreground element. In some examples, the identified portion of the image may be selected based on a semantic analysis of the image 310 indicating that the identified portion 350 of the image includes a depiction of a background element. The semantic analysis may identify a subject of interest. For example, the identified portion 350 of the image may be selected based on a semantic analysis of the image 310 indicating that the identified portion 350 of the image includes a depiction of a subject of interest. The semantic analysis may identify a region of interest. For example, the identified portion 350 of the image may be selected based on a semantic analysis of the image 310 indicating that the identified portion 350 of the image includes a region of interest. The semantic analysis may identify a scene shown in the image. For example, the identified portion 350 of the image may be selected based on a semantic analysis of the image 310 indicating that the identified portion 350 of the image includes a depiction of a particular portion of a scene shown in the image.

[0063]

[0072] In some examples, the digitally zoomed portion of the image may be identified based on a combination of one or more inputs from an input interface (e.g., touch input 335 of FIG. 3A) and one or more inputs from image analysis (e.g., face detection algorithm 355 of FIG. 3B, an object detection algorithm, a feature detection algorithm, semantic analysis as described above, or a combination thereof). For example, the image analysis may identify that the input from the input interface is near a depiction of a face in the image 310. The selection interface may center the digitally zoomed portion of the image on a point along a line between the center of the input from the input interface and the center of a bounding box that includes the detected depiction of the face in the image 310. For example, the point may be at a midpoint along the line.

[0064]

[0073] 4A is a conceptual diagram 400A showing an image 410 and a corresponding sharpness graph 415. The image 410 shows a bird. One side of the bird's head is shown in the image 410. One of the bird's eyes is shown in the image 410. A square is overlaid on the image 410 around the bird's eye, representing an identified portion 450 of the image 410. The identified portion 450 of the image 410 may be identified by a selection interface based on input from an input interface (e.g., a touch input on the bird's eye), based on image analysis (e.g., an object detector that identifies the bird's eye), or based on a combination thereof.

[0065]

[0074] In the context of Figures 4A-4C, a sharpness score is used as the image characteristic score. In particular, in Figures 4A-4C, an under-sharpness value is used as the image characteristic score. Image 410 has a high image characteristic score of 100%, i.e., a high under-sharpening value 455. Sharpness graph 415 shows the sharpness scoring of image 410. Sharpness graph 415 includes a vertical axis along which sharpness graph 415 shows the results of a modulation transfer function (MTF). The results of the MTF may be referred to as modulation, characteristic score, sharpness, percentage under-sharpening, contrast, spatial frequency response, or some combination thereof. In some examples, the MTF may be calculated according to the formula: modulation = (I max -I min ) / (I max +I min ) can be distinguished based on the formula: I max can represent the maximum intensity, and I min may represent a minimum intensity. The sharpness graph 415 includes a horizontal axis along which the sharpness graph 415 shows cycles per pixel. The cycles per pixel may represent how many cycles of maximum intensity lines (e.g., black lines) alternating with minimum intensity lines (e.g., white lines) are discernible (or would be discernible) for each pixel at the current level of sharpness (e.g., attribute score).

[0066]

[0075] FIG. 4B is a conceptual diagram 400B illustrating a first digitally zoomed portion 420 of the image 410 of FIG. 4A and a corresponding sharpness graph 425. The first digitally zoomed portion 420 of the image 410 is a cropped and resized (e.g., upscaled and / or upsampled) portion of the image 410 of FIG. 4A according to a first zoom intensity. The first digitally zoomed portion 420 of the image 410 is centered on an identified portion 450 of the image. The first digitally zoomed portion 420 of the image 410 has a high quality score of 100%, i.e., a high undersharpening value 460. The sharpness graph 425 illustrates the sharpness scoring of the image 410.

[0067]

[0076] 4C is a conceptual diagram 400C illustrating a second digitally zoomed portion 430 of the image 410 of FIG. 4A and a corresponding sharpness graph 435. In some examples, the second digitally zoomed portion 430 of the image 410 is a cropped and resized (e.g., upscaled and / or upsampled) portion of the image 410 of FIG. 4A according to a second zoom intensity that is stronger than the first zoom intensity used in FIG. 4B. In some examples, the second digitally zoomed portion 430 of the image 410 is a cropped and resized (e.g., upscaled and / or upsampled) portion of the first digitally zoomed portion 420 of the image 410 of FIG. 4B.

[0068]

[0077] The second digitally zoomed portion 430 of the image 410 includes the identified portion of the image 450. The second digitally zoomed portion 430 of the image 410 is vertically centered on the identified portion of the image 450. The second digitally zoomed portion 430 of the image 410 is not horizontally centered on the identified portion of the image 450, with the identified portion of the image 450 being slightly left of center in the second digitally zoomed portion 430 of the image 410. The second digitally zoomed portion 430 of the image 410 has a lower quality score of 72%, i.e., a lower undersharpening value 465. The sharpness graph 435 shows the sharpness scoring of the image 410.

[0069]

[0078] To preserve image sharpness, the image processing device may output the version of the image 410 with the highest zoom intensity (the most zoomed in version of the image 410) that still has a property score above a predefined property threshold, where the property score for the image 410 in FIG. 4A is 100%, the property score for the first digitally zoomed portion 420 of the image 410 in FIG. 4B is 100%, and the property score for the second digitally zoomed portion 430 of the image 410 in FIG. 4C is 72%. If the predefined property threshold is above 72% (e.g., 75%, 80%, 85%, 90%, 95%, 99%), the first digitally zoomed portion 420 of the image 410 in FIG. 4B will be output by the image processing device since its 100% property score exceeds the predefined property threshold. If the predefined characteristic threshold is below 72% (e.g., 70%, 65%, 60%, 55%, 50%), the second digitally zoomed portion 430 of the image 410 in FIG. 4C may be output by the image processing device since 72% of its characteristic scores exceed the predefined characteristic threshold.

[0070]

[0079] In the context of Figures 4A-4C, the characteristic score is generated based on a sharpness analysis (e.g., as described with respect to image characterization engine 235) of an identified portion 450 of image 410. The identified portion 450 of image 410 is included in image 410 of Figure 4A, the first digitally zoomed portion 420 of image 410 of Figure 4B, and the second digitally zoomed portion 430 of image 410 of Figure 4C. Thus, a characteristic score based on the identified portion 450 of image 410 as shown in each of them can provide a direct comparison between image 410 of Figure 4A, the first digitally zoomed portion 420 of image 410 of Figure 4B, and the second digitally zoomed portion 430 of image 410 of Figure 4C.

[0071]

[0080] In some examples, the characteristic score for the image 410 of Figure 4A may be generated based on a sharpness analysis of the entire image 410 of Figure 4A. In some examples, the characteristic score for the first digitally zoomed portion 420 of the image 410 of Figure 4B may be generated based on a sharpness analysis of the entire first digitally zoomed portion 420 of the image 410 of Figure 4B. In some examples, the characteristic score for the second digitally zoomed portion 430 of the image 410 of Figure 4C may be generated based on a sharpness analysis of the entire second digitally zoomed portion 430 of the image 410 of Figure 4C.

[0072]

[0081] 5 is a conceptual diagram 500 illustrating different zoom strengths and corresponding attribute scores along a series of incremental zoom operations. An image processing system receives an image captured by an image sensor. A first zoom strength of 1X 510 for the image corresponds to the image received from the image sensor. The image processing system performs a sharpness analysis of the image at the first zoom strength of 1X 510 and generates a first attribute score of "very high" 515.

[0073]

[0082] The image processing system performs a digital zoom-in operation on the original image to produce a zoomed image at a second zoom strength of 2X 520. The image processing system performs a sharpness analysis of the zoomed image at the second zoom strength of 2X 520 to generate a second attribute score 525 of "high."

[0074]

[0083] In some examples, the image processing system performs a second digital zoom-in operation to zoom in by an additional factor of 2X on the zoomed image at the second zoom strength 520 of 2X, producing a zoomed image at a third zoom strength 530 of 4X in total. In some examples, the image processing system may generate the zoomed image at the third zoom strength 530 of 4X directly from the original image by applying the third zoom strength 530 of 4X directly to the original image. The image processing system performs a sharpness analysis of the zoomed image at the third zoom strength 530 of 4X and generates a third characteristic score 535 of "medium".

[0075]

[0084] In some examples, the image processing system performs a third digital zoom-in operation to zoom in by an additional factor of 2X on the zoomed image at the third zoom strength 530 of 4X to produce a zoomed image at a total fourth zoom strength 540 of 8X. In some examples, the image processing system may generate the zoomed image at the fourth zoom strength 540 of 8X directly from the original image by applying the total fourth zoom strength 540 of 8X directly to the original image. The image processing system performs a sharpness analysis of the zoomed image at the fourth zoom strength 540 of 8X and generates a fourth characteristic score 545 of "low".

[0076]

[0085] In some examples, the image processing system performs a fourth digital zoom-in operation to zoom in by an additional factor of 2X on the zoomed image at the fourth zoom strength 540 of 8X, producing a zoomed image at a fifth zoom strength 550 of 16X in total. In some examples, the image processing system may generate the zoomed image at the fifth zoom strength 550 of 16X directly from the original image by applying the fifth zoom strength 550 of 16X directly to the original image. The image processing system performs a sharpness analysis of the zoomed image at the fifth zoom strength 550 of 16X and generates a fifth characteristic score 555 of "very low."

[0077]

[0086] In some examples, the image processing system identifies that the fourth characteristic score 545 is less than the predetermined characteristic threshold 560. For example, the predetermined characteristic threshold 560 may be "medium." The third characteristic score 535 is "medium" and therefore equal to the predetermined characteristic threshold 560. In some examples, the image processing system may determine that the image processing system outputs as its output image a zoomed image at a third zoom intensity 530 of 4X based on the third characteristic score 535 ("medium") meeting or exceeding the predetermined characteristic threshold 560 ("medium"). In some examples, meeting the predetermined characteristic threshold 560 may be insufficient and the image processing system may require a characteristic score to exceed the image processing system. In such examples, the image processing system may determine that the image processing system outputs as its output image a zoomed image at a second zoom intensity 520 of 2X based on the second characteristic score 525 ("high") meeting or exceeding the predetermined characteristic threshold 560 ("medium").

[0078]

[0087] 6 is a flow diagram 600 illustrating operations for processing image data. In some examples, the operations of the image processing technique illustrated in flow diagram 600 may be performed by an image processing system. In some examples, the image processing system is image processing system 200. In some examples, the classification system includes at least one of image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, image processing system 200, image sensor 205, ISP 210, post-processing engine 215, selection interface 230, image characterization engine 235, media encoder 240, selection interface associated with FIGS. 3A-3B, image processing system associated with FIGS. 4A-4C, trained ML model, trained NN, one or more network servers of a cloud service, computing system 700, or combinations thereof.

[0079]

[0088] In operation 605, the image processing system receives an image from an image sensor. The image may be captured by an image sensor of the image processing system. Examples of image sensors of operation 605 include image sensor 130 of FIG. 1 and image sensor 205 of FIG. 2. In some examples, the image is at least partially processed (e.g., by ISP 154 and / or ISP 210) before the image processing system receives the image. In some examples, the image processing system may include a connector coupled to the image sensor, and the image may be received using the connector. The connector may include a port, a jack, a wire, an input / output (IO) pin, a conductive trace on a printed circuit board (PCB), any other type of connector described herein, or some combination thereof. In some examples, the image processing system may include an image sensor that captures the image.

[0080]

[0089] In operation 610, the image processing system determines a first image characteristic score for a first set of image data. The first set of image data corresponds to a first zoomed version of the image at a first zoom intensity. In some examples, the image processing system generates the first zoomed version of the image, at least in part, by cropping and upscaling the image according to the first zoom intensity. The first digitally zoomed portion 420 of the image 410 of FIG. 4B is an example of a first zoomed version of the image (and / or the first set of image data) at a first zoom intensity.

[0081]

[0090] In operation 615, the image processing system determines a second image characteristic score for the second set of image data. The second set of image data corresponds to a second zoomed version of the image at a second zoom intensity. In some examples, the image processing system generates the second zoomed version of the image at least in part by cropping and upscaling at least one of the image and the first zoomed version of the image according to the second zoom intensity. The second digitally zoomed portion 430 of the image 410 of FIG. 4C is an example of the second zoomed version of the image (and / or the second set of image data) at the first zoom intensity.

[0082]

[0091] In some examples, the first set of image data corresponds to a first digitally zoomed portion of the image at a first zoom intensity. The first digitally zoomed portion 420 of the image 410 of FIG. 4B is an example of a first digitally zoomed portion of the image (and / or a first set of image data) at a first zoom intensity. In some examples, the second set of image data corresponds to a second digitally zoomed portion of the image at a second zoom intensity. The second digitally zoomed portion 430 of the image 410 of FIG. 4C is an example of a second digitally zoomed portion of the image (and / or a second set of image data) at a first zoom intensity. In some examples, the second digitally zoomed portion of the image is a subset of the first digitally zoomed portion of the image. For example, the second digitally zoomed portion 430 of FIG. 4C is a subset of the first digitally zoomed portion 420 of the image 410 of FIG. 4B. In some examples, the first digitally zoomed portion of the image is different from the second digitally zoomed portion of the image. For example, the second digitally zoomed portion 430 of FIG. 4C differs from the first digitally zoomed portion 420 of the image 410 of FIG. 4B.

[0083]

[0092] In some examples, generating the first digitally zoomed portion of the image at the first zoom intensity may include cropping the image. In some examples, generating the second digitally zoomed portion of the image at the second zoom intensity may include cropping the image or the first digitally zoomed portion of the image. In some examples, generating the first digitally zoomed portion of the image at the first zoom intensity may include upscaling and / or upsampling the image. In some examples, generating the second digitally zoomed portion of the image at the second zoom intensity may include upscaling and / or upsampling the first digitally zoomed portion of the image.

[0084]

[0093] In some examples, the image sensor may be a high-resolution image sensor. The image may be pixel-binned as it is received from the image sensor in operation 605. In pixel-binning, values ​​(e.g., charge values) for one or more adjacent pixels (e.g., a square of four pixels) may be averaged or added and reported as a single superpixel. Pixel-binning may reduce noise. In some examples, rather than upscaling and / or upsampling the image to generate the first digitally zoomed portion of the image and / or the second digitally zoomed portion of the image, the image processing system may disable binning and use original pixel values ​​from the image sensor for the zoomed image(s) rather than the superpixels. In some examples, rather than upscaling and / or upsampling the image to generate a first digitally zoomed portion of the image and / or a second digitally zoomed portion of the image, the image processing system may reduce binning by combining fewer original pixels for each superpixel and may use reduced-binning super pixel values ​​for the zoomed image(s).

[0085]

[0094] In operation 620, the image processing system identifies that the second image characteristic score is less than the image characteristic threshold value. In some examples, the image processing system also identifies that the first image characteristic score (of operation 610) is greater than or equal to the image characteristic threshold value.

[0086]

[0095] In some examples, the first image characteristic score is a first image sharpness score, the second image characteristic score is a second image sharpness score, and the image characteristic threshold is an image sharpness threshold. In some examples, the first image characteristic score is a first undersharpening value of the first set of image data, the second image characteristic score is a second undersharpening value of the second set of image data, and the image characteristic threshold is an undersharpening value threshold.

[0087]

[0096] In an illustrative example, the image characteristic threshold of operation 615 may be image characteristic threshold 560 of FIG. 5. The second image characteristic score of operation 615 and operation 620 may be image characteristic score 545 (low) or image characteristic score 555 (very low) of FIG. 5. Thus, the second zoomed version of the image at the second zoom intensity may be a zoomed version of the image at zoom intensity 540 (8X) or at zoom intensity 550 (16X) of FIG. 5. The first image characteristic score of operation 610 may be image characteristic score 515 (very high), image characteristic score 525 (high), or image characteristic score 535 (medium) of FIG. 5. Thus, the first zoomed version of the image at the first zoom intensity may be a zoomed version of the image at zoom intensity 510 (1X), at zoom intensity 520 (2X), or at zoom intensity 530 (4X) of FIG. 5.

[0088]

[0097] In operation 625, the image processing system outputs the first set of image data as an output image based on identifying the second image characteristic score being less than the image characteristic threshold value. In some examples, outputting the first set of image data as an output image is also based on the first image characteristic score being greater than or equal to the image characteristic threshold value.

[0089]

[0098] In some examples, the first set of image data and / or the output image and / or the first zoomed version of the image at the first zoom intensity may be a zoomed version of the image at zoom intensity 510 (1X), at zoom intensity 520 (2X), or at zoom intensity 530 (4X) of Figure 5. In an illustrative example, the first set of image data and / or the output image and / or the first zoomed version of the image at the first zoom intensity may be a zoomed version of the image at zoom intensity 530 (4X) of Figure 5 because zoom intensity 530 (4X) is the highest possible zoom intensity for which the corresponding image characteristic score (image characteristic score for "medium" 535) is equal to or greater than image characteristic threshold 560 ("medium") of Figure 5.

[0090]

[0099] In some examples, to output the first set of image data as an output image, the image processing system is configured to and can render the output image for display on a display screen. In some examples, to output the first set of image data as an output image, the image processing system is configured to and can display the output image on a display screen. The image processing system can include a display screen. Examples of display screens can include I / O 156, I / O 160, display screen 225, output device 735, or a combination thereof. In some examples, to output the first set of image data as an output image, the image processing system is configured to and can encode the output image for display on a display screen. The encoding and / or rendering of the output image can be performed by host processor 152, image processor 150, post-processing engine 215, camera preview engine 220, media encoder 240, output device 735, or a combination thereof.

[0091]

[0100] In some examples, to output the first set of image data as an output image, the one or more processors are configured to transmit the output image using a communication interface. The image processing system may include a communication interface that may be configured to transmit and may transmit the output image to a device. Examples of communication interfaces may include I / O 156, I / O 160, output device 735, communication interface 740, or a combination thereof.

[0092]

[0101] In some examples, the image processing system also receives one or more inputs identifying a portion of the image, where the first zoomed version of the image and the second zoomed version of the image are based on the identified portion of the image. Examples of the identified portion of the image include the identified portion 330 of the image 310 of Figure 3A, the digitally zoomed portion 320 of the image 310 of Figure 3A, the identified portion 350 of the image 310 of Figure 3B, the digitally zoomed portion 340 of the image 310 of Figure 3B, the identified portion 450 of the image 410 of Figures 4A-4C, the first digitally zoomed portion 420 of the image 410 of Figure 4B, the second digitally zoomed portion 430 of the image 410 of Figure 4C, or a combination thereof.

[0093]

[0102] In some examples, the first zoomed version of the image and the second zoomed version of the image include the identified portion of the image. In some examples, the one or more inputs include a user input through a user interface. In some examples, the user input and / or the one or more inputs include at least one of a touch input, a hover input, a gesture input, a gaze input, a button press, a pointer movement, a pointer click, or a combination thereof. An example of the user input and / or the one or more inputs includes the touch input 335 of FIG. 3A. In some examples, the image processing system receives one or more inputs from an object detection algorithm, and the object detection algorithm determines that the identified portion of the image includes a depiction of a type of object. In some examples, the type of object is a face. An example of an object detection algorithm includes the face detection algorithm 355 of FIG. 3B. The object detection algorithm may be a face detection algorithm, a face recognition algorithm, a face tracking algorithm, an object detection algorithm, an object recognition algorithm, an object tracking algorithm, a feature detection algorithm, a feature recognition algorithm, a feature tracking algorithm, a semantic analysis algorithm, or a combination thereof.

[0094]

[0103] In some examples, the image processing system can receive a first sensor readout corresponding to a first portion of a scene. The image processing system can evaluate the readout. The image processing system can receive a second sensor readout corresponding to a second portion of the scene at a different zoom strength. The image processing system can select one of the first sensor readout or the second sensor readout and output as an output image. In some examples, the image processing system can determine a first image characteristic score for the first sensor readout and a second image characteristic score for the second sensor readout. The image processing system can select the first sensor readout as the output image based on, for example, determining that the first image characteristic score is higher than the second image characteristic score. The image processing system can select the first sensor readout as the output image based on, for example, determining that the first image characteristic score is higher than an image characteristic threshold and / or the second image characteristic score is lower than an image characteristic threshold.

[0095]

[0104] The image processing technique illustrated in flowchart 600 may include any operations described or illustrated in or described with respect to any of the conceptual diagrams, block diagrams, and flowcharts 100, 200, 300, 500, and / or 700.

[0096]

[0105] In some cases, at least a subset of the techniques illustrated by any of the conceptual diagrams, block diagrams, and flow diagrams 100, 200, 300, 500, and / or 700 may be performed remotely by one or more network servers of a cloud service. In some examples, the processes described herein (e.g., processes including those illustrated by the conceptual diagrams, block diagrams, and flow diagrams 200, 300, 500, 600, 1000, 1100, 1200, 600, and / or other processes described herein) may be performed by a computing system or apparatus. In some examples, the processes illustrated by the conceptual diagrams, block diagrams, and flow diagrams 100, 200, 300, 500, and / or 700 may be performed by the image capture device 105A of FIG. 1, the image processing device 105B of FIG. 1, and / or the image capture and processing system 100 of FIG. 1. In some examples, the processes illustrated by the conceptual diagrams, block diagrams, and flowcharts 100, 200, 300, 500, and / or 700 may be implemented by the image processing system 200 of Figure 2. In some examples, the processes illustrated by the conceptual diagrams, block diagrams, and flowcharts 100, 200, 300, 500, and / or 700 may be implemented by a computing system having the architecture of computing system 700 shown in Figure 7. The computing system may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a networked watch or smart watch, or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device with resource capabilities to implement the processes described herein, including the processes illustrated by the conceptual diagrams, block diagrams, and flowcharts 100, 200, 300, 500, and / or 700.In some cases, a computing system or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other component(s) configured to perform steps of processes described herein. In some examples, a computing system may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other component(s). The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0097]

[0106] Components of a computing system may be implemented in circuits. For example, components may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry) and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0098]

[0107] The processes illustrated by the conceptual, block, and flow diagrams 100, 200, 300, 500, and / or 700 are organized as logical flow diagrams whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described is not to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0099]

[0108] Additionally, the processes illustrated by the conceptual diagrams, block diagrams, and flow diagrams 100, 200, 300, 500, 700, and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored in a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0100]

[0109] FIG. 7 illustrates an example of a system for implementing some aspects of the present technology. In particular, FIG. 7 illustrates an example of a computing system 700, which may be any computing device or system constituting, for example, the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image processor 150, the host processor 152, the ISP 154, the image processing system 200, the image sensor 205, the ISP 210, the post-processing engine 215, the selection interface 230, the image characterization engine 235, the media encoder 240, the selection interface associated with FIG. 3A-3B, the image processing system associated with FIG. 4A-4C, one or more trained ML models, one or more trained NNs, one or more network servers, a camera, any combination thereof, or any components thereof, in which the components of the system communicate with each other using connections 705. The connections 705 may be physical connections to the processor 710 using a bus, or direct connections to the processor 710, such as in a chipset architecture. The connections 705 may also be virtual connections, networked connections, or logical connections.

[0101]

[0110] In some embodiments, computing system 700 is a distributed system in which the functionality described in this disclosure may be distributed within a data center, multiple data centers, a peer network, etc. In some embodiments, one or more of the system components described represent many such components, each of which performs some or all of the functionality described for it. In some embodiments, a component may be a physical or virtual device.

[0102]

[0111] The exemplary system 700 includes at least one processing unit (CPU or processor) 710 and a connection 705 that couples various system components, including system memory 715, such as read only memory (ROM) 720 and random access memory (RAM) 725, to the processor 710. The computing system 700 may include a cache 712 of high-speed memory directly connected to the processor 710, in close proximity to the processor 710, or integrated as part of the processor 710.

[0103]

[0112] Processor 710 may include any general purpose processor and hardware or software services, such as services 732, 734, and 736 stored in storage device 730, configured to control processor 710 as well as special purpose processors, where software instructions are incorporated into the actual processor design. Processor 710 may essentially be a fully self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0104]

[0113] To enable user interaction, computing system 700 includes input device(s) 745, which may represent any number of input mechanisms, such as a microphone for audio, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Computing system 700 may also include output device(s) 735, which may be one or more of several output mechanisms. In some instances, a multimodal system may allow a user to provide multiple types of input and output for communicating with computing system 700. Computing system 700 may include a communication interface 740, which may generally govern and manage user input and system output.The communications interface may be an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet® port / plug, a fiber optic port / plug, a proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communications (DSRC) wireless signal transmission, 802.11 Wi-Fi® wireless signal transmission, a wireless local area network (WLAN) ... The communication interface 740 may implement or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those utilizing wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof. The communication interface 740 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine a location of the computing system 700 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS.There is no restriction to operating on any particular hardware configuration, and therefore the basic features herein can be easily substituted for improved hardware or firmware configurations as they are developed.

[0105]

[0114] The storage device 730 may be a non-volatile and / or non-transitory and / or computer readable memory device, and may be a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disk read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a blu-ray disk (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory The memory may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer, such as a Stick card, a smart card chip, an EMV chip, a Subscriber Identity Module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0106]

[0115] Storage devices 730 may include software services, servers, services, etc., which cause the system to perform functions when code defining such software is executed by processor 710. In some embodiments, hardware services that perform a particular function may include software components stored in a computer-readable medium in relation to the necessary hardware components, such as processor 710, connections 705, output devices 735, etc., to perform that function.

[0107]

[0116] The term "computer-readable medium" as used herein includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instruction(s) and / or data. Computer-readable media may include non-transitory media on which data may be stored and does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0108]

[0117] In some embodiments, computer-readable storage devices, media, and memories may include cable or wireless signals containing bit streams, etc. However, when stated, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0109]

[0118] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will appreciate that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a method implemented in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0110]

[0119] Individual embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process is terminated when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0111]

[0120] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general purpose computer, a special purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0112]

[0121] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Common examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in peripheral devices or add-in cards. Such functionality may also be implemented on a circuit board among different chips or different processes executing in a single device, as further examples.

[0113]

[0122] The instructions, media for carrying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0114]

[0123] In the above description, aspects of the present application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various ways, and the appended claims are to be construed to include such variations, except as limited by the prior art. Various features and aspects of the applications described above may be used individually or together. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of explanation, the methods have been described in a particular order. It should be appreciated that in alternative embodiments, the methods may be performed in an order different from that described.

[0115]

[0124] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terminology used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0116]

[0125] When a component is described as being "configured to" perform some operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0117]

[0126] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component, and / or any component that is in communication, either directly or indirectly, with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communications interface).

[0118]

[0127] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or more than one member of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" can mean A, B, or A and B, and can further include items not listed in the set of A and B.

[0119]

[0128] The various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0120]

[0129] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Features described as modules or components may be implemented together in an integrated logic device, or separately as separate but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or radio wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0121]

[0130] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or may be incorporated into a composite video encoder / decoder (codec).

[0122]

[0131] Exemplary aspects of the present disclosure include the following.

[0123]

[0132] Aspect 1: An apparatus for processing image data, the apparatus comprising: a memory; and one or more processors coupled to the memory, the one or more processors configured to: receive an image from an image sensor; determine a first image characteristic score for a first set of image data; determine a second image characteristic score for a second set of image data, the first set of image data corresponding to a first zoomed variant of the image at a first zoom intensity; identify that the second image characteristic score is less than an image characteristic threshold, the second set of image data corresponding to a second zoomed variant of the image at a second zoom intensity; and output the first set of image data as an output image based on identifying that the second image characteristic score is less than the image characteristic threshold.

[0124]

[0133] Aspect 2: The apparatus of aspect 1, wherein outputting the first set of image data as the output image is also based on the first image characteristic score being greater than or equal to an image characteristic threshold value.

[0125]

[0134] Aspect 3: An apparatus described in any of aspects 1 to 2, wherein the first image characteristic score is a first image sharpness score, the second image characteristic score is a second image sharpness score, and the image characteristic threshold is an image sharpness threshold.

[0126]

[0135] Aspect 4: An apparatus described in any of aspects 1 to 3, wherein the first image characteristic score is a first undersharpening value of a first set of image data, the second image characteristic score is a second undersharpening value of a second set of image data, and the image characteristic threshold is an undersharpening value threshold.

[0127]

[0136] Aspect 5: An apparatus as described in any of aspects 1 to 4, wherein one or more processors are configured to receive one or more inputs identifying a portion of the image, and wherein a first zoomed version of the image and a second zoomed version of the image are based on the identified portion of the image.

[0128]

[0137] Example 6: The apparatus of example 5, wherein the first zoomed version of the image and the second zoomed version of the image include the identified portion of the image.

[0129]

[0138] Example 7: The apparatus of any of examples 5 or 6, wherein the one or more inputs include user input through a user interface.

[0130]

[0139] Example 8: A device described in any of examples 5 to 7, wherein the one or more inputs include at least one of touch input, gesture input, and gaze input.

[0131]

[0140] Aspect 9: An apparatus described in any of aspects 5 to 8, wherein the one or more processors are configured to receive one or more inputs from an object detection algorithm, the object detection algorithm determining that the identified portion of the image includes a depiction of a type of object.

[0132]

[0141] Example 10: The apparatus of example 9, wherein the object type is a face.

[0133]

[0142] Aspect 11: An apparatus as described in any of aspects 1 to 10, wherein the one or more processors are configured to generate a first zoomed version of the image, at least in part, by cropping and upscaling the image according to a first zoom intensity, and to generate a second zoomed version of the image, at least in part, by cropping and upscaling at least one of the image and the first zoomed version of the image according to a second zoom intensity.

[0134]

[0143] Aspect 12: An apparatus as described in any of aspects 1 to 11, wherein to output the first set of image data as an output image, one or more processors are configured to render the output image for display on a display screen.

[0135]

[0144] Example 13: The apparatus of any of Examples 1 to 12, further comprising a display screen configured to display the output image.

[0136]

[0145] Aspect 14: An apparatus described in any of aspects 1 to 13, wherein the one or more processors are configured to identify that a difference between the first image characteristic score and the second image characteristic score is less than a difference threshold, and output a second set of image data as a second output image based on the difference being less than the difference threshold.

[0137]

[0146] Aspect 15: An apparatus described in any of aspects 1 to 14, wherein the first set of image data corresponds to a first digitally zoomed portion of the image at a first zoom intensity, and the second set of image data corresponds to a second digitally zoomed portion of the image at a second zoom intensity.

[0138]

[0147] Aspect 16: The apparatus of aspect 15, wherein the second digitally zoomed portion of the image is a subset of the first digitally zoomed portion of the image.

[0139]

[0148] Example 17: The apparatus of example 15, wherein the first digitally zoomed portion of the image is different from the second digitally zoomed portion of the image.

[0140]

[0149] Aspect 18: An apparatus described in any of aspects 1 to 17, wherein the one or more processors are configured to transmit the output image using the communications interface to output the first set of image data as an output image.

[0141]

[0150] Example 19: An apparatus described in any of examples 1 to 18, further comprising a communication interface configured to transmit the output image to a device.

[0142]

[0151] Aspect 20: An apparatus as described in any of aspects 1 to 19, wherein the apparatus is one of a mobile device, a mobile handset, a wireless communication device, a head mounted display, and a camera.

[0143]

[0152] Example 21: An apparatus described in any of examples 1 to 20, further comprising an image sensor configured to capture an image.

[0144]

[0153] Aspect 22: A method for processing image data, the method comprising: receiving an image from an image sensor; determining a first image characteristic score for a first set of image data; determining a second image characteristic score for a second set of image data, the first set of image data corresponding to a first zoomed variant of the image at a first zoom intensity; identifying that the second image characteristic score is less than an image characteristic threshold, the second set of image data corresponding to a second zoomed variant of the image at a second zoom intensity; and outputting the first set of image data as an output image based on identifying that the second image characteristic score is less than the image characteristic threshold.

[0145]

[0154] Aspect 23: The method of aspect 22, wherein outputting the first set of image data as the output image is also based on the first image characteristic score being greater than or equal to an image characteristic threshold value.

[0146]

[0155] Example 24: A method according to any of examples 22 to 23, wherein the first image characteristic score is a first image sharpness score, the second image characteristic score is a second image sharpness score, and the image characteristic threshold is an image sharpness threshold.

[0147]

[0156] Aspect 25: A method according to any of aspects 22 to 24, wherein the first image characteristic score is a first undersharpening value of a first set of image data, the second image characteristic score is a second undersharpening value of a second set of image data, and the image characteristic threshold is an undersharpening value threshold.

[0148]

[0157] Example 26: The method of any of examples 22 to 25, further comprising receiving one or more inputs identifying portions of the image, wherein the first zoomed variant of the image and the second zoomed variant of the image are based on the identified portions of the image.

[0149]

[0158] Example 27: The method of example 26, wherein the first zoomed version of the image and the second zoomed version of the image include the identified portion of the image.

[0150]

[0159] Aspect 28: The method of any of aspects 26 or 27, wherein the one or more inputs include user input through a user interface.

[0151]

[0160] Aspect 29: A method as described in any of aspects 26 to 28, wherein the one or more inputs include at least one of touch input, gesture input, and gaze input.

[0152]

[0161] Aspect 30: A method as described in any of aspects 26 to 29, further comprising receiving one or more inputs from an object detection algorithm, the object detection algorithm determining that the identified portion of the image includes a depiction of a type of object.

[0153]

[0162] Aspect 31: The method described in aspect 30, wherein the object type is a face.

[0154]

[0163] Aspect 32: A method as described in any of aspects 22 to 31, wherein one or more processors are configured to generate a first zoomed version of the image, at least in part, by cropping and upscaling the image according to a first zoom intensity, and generate a second zoomed version of the image, at least in part, by cropping and upscaling at least one of the image and the first zoomed version of the image according to a second zoom intensity.

[0155]

[0164] Aspect 33: The method of any of aspects 22 to 32, wherein outputting the first set of image data as the output image includes rendering the output image for display on a display screen.

[0156]

[0165] Aspect 34: A method described in any of aspects 22 to 33, wherein the one or more processors are configured to identify that a difference between the first image characteristic score and the second image characteristic score is less than a difference threshold, and output a second set of image data as a second output image based on the difference being less than the difference threshold.

[0157]

[0166] Aspect 35: A method as described in any of aspects 22 to 34, wherein the first set of image data corresponds to a first digitally zoomed portion of the image at a first zoom intensity, and the second set of image data corresponds to a second digitally zoomed portion of the image at a second zoom intensity.

[0158]

[0167] Aspect 36: The method of aspect 35, wherein the second digitally zoomed portion of the image is a subset of the first digitally zoomed portion of the image.

[0159]

[0168] Example 37: The method of example 35, wherein the first digitally zoomed portion of the image is different from the second digitally zoomed portion of the image.

[0160]

[0169] Aspect 38: The method of any of aspects 22 to 37, wherein outputting the first set of image data as the output image includes transmitting the output image using a communications interface.

[0161]

[0170] Aspect 39: A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform any of the operations recited in 2 to 38.

[0162]

[0171] Embodiment 40: An apparatus for image processing, comprising means for performing the operations described in any of embodiments 2 to 38.

Claims

1. 1. An apparatus for processing image data, the apparatus comprising: Memory, one or more processors coupled to the memory; wherein the one or more processors: receiving an image from an image sensor; determining a first image characteristic score for a first set of image data, the first set of image data corresponding to a first zoomed version of the image at a first zoom intensity; determining a second image characteristic score for a second set of image data, the second set of image data corresponding to a second zoomed version of the image at a second zoom intensity; Identifying the second image characteristic score being less than an image characteristic threshold; outputting the first set of image data as an output image based on identifying the second image characteristic score being less than the image characteristic threshold; wherein the first image characteristic score comprises a first image sharpness score, the second image characteristic score comprises a second image sharpness score, and the image characteristic threshold comprises an image sharpness threshold.

2. The apparatus of claim 1 , wherein outputting the first set of image data as the output image is also based on the first image characteristic score being greater than or equal to the image characteristic threshold value.

3. 2. The apparatus of claim 1, wherein the first image characteristic score is a first undersharpened value of the first set of image data, the second image characteristic score is a second undersharpened value of the second set of image data, and the image characteristic threshold is an undersharpened value threshold.

4. The one or more processors: receiving one or more inputs identifying a portion of the image, wherein the first zoomed version of the image and the second zoomed version of the image are based on the identified portion of the image.

2. The apparatus of claim 1.

5. The apparatus of claim 4 , wherein the first zoomed version of the image and the second zoomed version of the image include the identified portion of the image.

6. The apparatus of claim 4 , wherein the one or more inputs include a user input through a user interface.

7. The device of claim 6 , wherein the one or more inputs include at least one of a touch input, a gesture input, and a gaze input.

8. 5. The apparatus of claim 4, wherein the one or more processors are configured to receive the one or more inputs from an object detection algorithm, the object detection algorithm determining that the identified portion of the image includes a depiction of a type of object.

9. The one or more processors: generating the first zoomed version of the image at least in part by cropping and upscaling the image according to the first zoom intensity; generating the second zoomed version of the image by at least partially cropping and upscaling at least one of the image and the first zoomed version of the image according to the second zoom intensity; The apparatus of claim 1 configured to:

10. 2. The apparatus of claim 1, wherein the first set of image data corresponds to a first digitally zoomed portion of the image at the first zoom intensity, and the second set of image data corresponds to a second digitally zoomed portion of the image at the second zoom intensity.

11. The apparatus of claim 10 , wherein the second digitally zoomed portion of the image is a subset of the first digitally zoomed portion of the image.

12. The apparatus of claim 10 , wherein the first digitally zoomed portion of the image is different from the second digitally zoomed portion of the image.

13. 1. A method for processing image data, the method comprising: receiving an image from an image sensor; determining a first image characteristic score for a first set of image data, the first set of image data corresponding to a first zoomed version of the image at a first zoom intensity; determining a second image characteristic score for a second set of image data, the second set of image data corresponding to a second zoomed version of the image at a second zoom intensity; Identifying the second image characteristic score being less than an image characteristic threshold; outputting the first set of image data as an output image based on identifying the second image characteristic score being less than the image characteristic threshold; wherein the first image characteristic score comprises a first image sharpness score, the second image characteristic score comprises a second image sharpness score, and the image characteristic threshold comprises an image sharpness threshold.

14. The method of claim 13 , wherein outputting the first set of image data as the output image is also based on the first image characteristic score being greater than or equal to the image characteristic threshold value.

15. 14. The method of claim 13, wherein the first image characteristic score is a first undersharpened value of the first set of image data, the second image characteristic score is a second undersharpened value of the second set of image data, and the image characteristic threshold is an undersharpened value threshold.

Images