Systems and methods for camera zoom

Abstract

Examples of digital zoom that maintain image characteristics such as sharpness, clarity, and / or contrast are described. In some aspects, the device can receive an image and determine various image characteristic scores corresponding to digital zoom variations with different zoom intensities. For example, the device can determine a first image characteristic score for a first zoom intensity and a second image characteristic score for a second zoom intensity. The device can compare the image characteristic scores to image characteristic thresholds and can select the highest zoom intensity corresponding to an image characteristic score not lower than the image characteristic threshold. For example, if the first image characteristic score meets or exceeds the image characteristic threshold while the second image characteristic score does not meet or exceeds the image characteristic threshold, the device can select the first zoom intensity. The device can output image data corresponding to the image at the selected zoom intensity.

Description

Technical Field

[0001] This application relates to image processing. More specifically, aspects of this application relate to systems and methods for identifying high-intensity or high-level digital zoom in an image, wherein the image maintains at least a threshold level one or more image characteristics, such as sharpness, contrast, or clarity. Background Technology

[0002] In optics, zoom refers to the technique that allows a camera to capture an image that appears to be captured from a subject closer to or farther away, depending on the zoom setting. Magnification refers to the adjustment that makes a captured image appear to be captured from a closer subject. Reduction refers to the adjustment that makes a captured image appear to be captured from a more distant subject. Zoom includes optical zoom and digital zoom. Optical zoom produces a sharp image but involves the physical movement of one or more lenses in the camera. Digital zoom traditionally involves cropping and enlarging a portion of the captured image. Digital zoom does not require any specialized camera equipment. However, digitally zooming an image at a high scaling intensity can negatively affect image characteristics such as the sharpness, contrast, and / or clarity of the digitally scaled portion of the image compared to the corresponding image characteristics of the original image. Summary of the Invention

[0003] This document describes systems and techniques for digital zoom. In some aspects, the device may receive an image captured by an image sensor. The device may determine various image characteristic scores corresponding to digitally zoomed versions of the image with different zoom intensities (also known as zoom levels). For example, the device may determine a first image characteristic score for a first zoom intensity (or zoom level) and a second image characteristic score for a second zoom intensity (or zoom level). The device may compare the image characteristic scores to an image characteristic threshold and may select the highest zoom intensity corresponding to an image characteristic score not lower than the image characteristic threshold. In some examples, the device may select the highest zoom intensity corresponding to an image characteristic score higher than the image characteristic threshold. In some examples, the device may select the highest zoom intensity corresponding to an image characteristic score equal to the image characteristic threshold. For example, the device may select the first zoom intensity if the first image characteristic score meets or exceeds the image characteristic threshold and the second image characteristic score is lower than the image characteristic threshold. The device may output image data corresponding to the digitally zoomed portion of the image at the selected zoom intensity (e.g., displaying image data, storing image data, sending image data to another device, etc.).

[0004] In one example, an apparatus for image processing is provided. The apparatus includes a memory and one or more processors (e.g., implemented in a circuit) coupled to the memory. The one or more processors are configured to: receive an image from an image sensor; determine a first image feature score for a first set of image data corresponding to a first scaling variation of the image at a first scaling intensity; determine a second image feature score for a second set of image data corresponding to a second scaling variation of the image at a second scaling intensity; identify that the second image feature score is less than an image feature threshold; and output the first set of image data as an output image based on the identification that the second image feature score is less than the image feature threshold.

[0005] In another example, an image processing method is provided. The method includes receiving an image captured by an image sensor. The method includes: receiving the image from the image sensor; determining a first image feature score for a first image dataset, the first image dataset corresponding to a first scaling variation of the image at a first scaling intensity; determining a second image feature score for a second image dataset, the second image dataset corresponding to a second scaling variation of the image at a second scaling intensity; identifying that the second image feature score is less than an image feature threshold; and outputting the first image dataset as an output image based on the identification that the second image feature score is less than the image feature threshold.

[0006] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: receive an image from an image sensor; determine a first image feature score for a first image dataset corresponding to a first scaling variant of the image at a first scaling intensity; determine a second image feature score for a second image dataset corresponding to a second scaling variant of the image at a second scaling intensity; identify that the second image feature score is less than an image feature threshold; and output the first image dataset as an output image based on the identification that the second image feature score is less than the image feature threshold.

[0007] In another example, an apparatus for image processing is provided. The apparatus includes: components for receiving images from an image sensor; components for determining a first image feature score for a first image dataset corresponding to a first scaling variation of the image at a first scaling intensity; components for determining a second image feature score for a second image dataset corresponding to a second scaling variation of the image at a second scaling intensity; components for identifying if the second image feature score is less than an image feature threshold; and components for outputting the first image dataset as an output image based on the identification that the second image feature score is less than the image feature threshold.

[0008] In some aspects, the output of the first image dataset is based on the output image also being based on the first image feature score being greater than or equal to the image feature threshold.

[0009] In some aspects, the first image feature score is a first image sharpness score, the second image feature score is a second image sharpness score, and the image feature threshold is an image sharpness threshold. In some aspects, the first image feature score is a first undersharpened value of the first image dataset, the second image feature score is a second undersharpened value of the second image dataset, and the image feature threshold is an undersharpened value threshold.

[0010] In some aspects, the methods, apparatus, and computer-readable media described above further include: receiving one or more inputs identifying a portion of the image, wherein the first scaling variant and the second scaling variant of the image are based on the identified portion of the image. In some aspects, the first scaling variant and the second scaling variant of the image include the identified portion of the image. In some aspects, the one or more inputs include user input via a user interface. In some aspects, the one or more inputs include at least one of touch input, hover input, gesture input, and gaze input. In some aspects, the methods, apparatus, and computer-readable media described above further include: receiving the one or more inputs from an object detection algorithm that determines that the identified portion of the image includes a depiction of an object type. In some aspects, the object type is a face.

[0011] In some aspects, the methods, apparatus, and computer-readable media described above further include: generating a first scaling variant of an image by cropping and enlarging the image according to a first scaling intensity at least in part; and generating a second scaling variant of an image by cropping and enlarging the image according to a second scaling intensity and at least one of the first scaling variant of the image.

[0012] In some aspects, in order to output the first image dataset as the output image, the one or more processors are configured to render the output image for display on a display screen. In some aspects, the device described above further includes the display screen configured to display the output image.

[0013] In some aspects, the methods, apparatus, and computer-readable media described above further include: identifying that the difference between a first image feature score and a second image feature score is less than a difference threshold; and outputting the second image dataset as a second output image based on the fact that the difference is less than the difference threshold.

[0014] In some aspects, the first image data set corresponds to a first digitally scaled portion of the image at a first scaling intensity, and the second image data set corresponds to a second digitally scaled portion of the image at a second scaling intensity. In some aspects, the second digitally scaled portion of the image is a subset of the first digitally scaled portion of the image. In some aspects, the first digitally scaled portion of the image is different from the second digitally scaled portion of the image.

[0015] In some aspects, in order to output the first image dataset as the output image, the one or more processors are configured to send the output image using a communication interface. In some aspects, the device described above further includes a communication interface configured to send the output image to the apparatus.

[0016] In some aspects, the device described above is one of a mobile device, a mobile handheld device, a wireless communication device, a head-mounted display, and a camera. In some aspects, the device described above further includes: the image sensor configured to capture the image.

[0017] In some aspects, the device includes a camera, mobile device, mobile phone, smartphone, handheld device, portable gaming device, wireless communication device, smartwatch, wearable device, head-mounted display (HMD), extended reality device (e.g., virtual reality (VR), augmented reality (AR), or mixed reality (MR) device), personal computer, laptop computer, server computer, or other device. In some aspects, one or more processors include an image signal processor (ISP). In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus includes an image sensor for capturing images. In some aspects, the apparatus also includes a display for displaying images, one or more notifications (e.g., associated with image processing), and / or other displayable data. In some aspects, the display shows the image after one or more processors have processed it.

[0018] This invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to define the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate portions of the entire specification, any or all of the drawings, and each claim.

[0019] The foregoing and other features and embodiments will become more apparent from the following description, claims and drawings. Attached Figure Description

[0020] The illustrative embodiments of this application are described in detail below with reference to the accompanying drawings:

[0021] Figure 1 This is a block diagram illustrating the architecture of an image capture and processing system;

[0022] Figure 2 This is a block diagram illustrating the architecture of an image processing system, including an image feature evaluation engine;

[0023] Figure 3A This is a conceptual diagram showing the selection interface, where a digitally scaled portion of the image is centered on a portion of the image based on touch input identification;

[0024] Figure 3B This is a conceptual diagram showing the selection interface, where a digitally scaled portion of the image is centered on a portion of the image identified based on image analysis.

[0025] Figure 4A It is a conceptual diagram showing an image and its corresponding sharpness curve;

[0026] Figure 4B It is shown Figure 4A A conceptual diagram of the first digitally scaled portion of the image and its corresponding sharpness curve;

[0027] Figure 4C It is shown Figure 4A A conceptual diagram of the second digitally scaled portion of the image and the corresponding sharpness curve;

[0028] Figure 5 This is a conceptual diagram illustrating different zoom intensities and corresponding feature scores along a series of incremental zoom operations;

[0029] Figure 6 This is a flowchart illustrating the operations used to process image data; and

[0030] Figure 7 This is a diagram illustrating an example of a system used to implement certain aspects of this technology. Detailed Implementation

[0031] Certain aspects and embodiments of this disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some can be applied in combination, as will be apparent to those skilled in the art. In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of embodiments of this application. However, it will be apparent that various embodiments can be practiced without these specific details. The drawings and description are not intended to be limiting.

[0032] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0033] An image capture device (e.g., a camera) is a device that receives light and captures image frames (such as still images or video frames) using an image sensor. The terms "image," "image frame," and "frame" are used interchangeably herein. An image capture device typically includes at least one lens that receives light from a scene and bends the light toward the image sensor of the image capture device. The light received by the lens passes through an aperture controlled by one or more control mechanisms and is received by the image sensor. One or more control mechanisms may control exposure, focus, and / or zoom based on information from the image sensor and / or based on information from an image processor (e.g., a host or application process and / or an image signal processor). In some examples, one or more control mechanisms include a motor or other control mechanism that moves the lens of the image capture device to a target lens position.

[0034] Zoom refers to a set of techniques that allow a camera to capture images that appear to be captured from a subject that is closer or farther away, depending on the zoom setting. Magnification refers to the adjustment that makes a captured image appear to be captured from a closer subject. Zoom reduction refers to the adjustment that makes a captured image appear to be captured from a farther subject. Zoom includes optical zoom and digital zoom.

[0035] Optical zoom is a technique that performs zooming by changing the focal length of a camera's zoom lens or by switching from a camera with a first focal length to a camera with a second focal length. Magnification through optical zoom does not degrade image quality, but it requires specialized camera equipment and must be performed before capturing the image.

[0036] Digital zoom is a technique used to enlarge a portion of an image by cropping and magnifying the cropped portion. Digital zoom can be beneficial because it doesn't require any specialized camera equipment, such as telephoto lenses, and therefore can be implemented on devices with fairly basic cameras. In addition to optical zoom, digital zoom can be used to magnify beyond the optical zoom capabilities of a particular camera. Digital zoom can be performed at any point after an image is captured. However, magnification using digital zoom involves cropping and enlarging a portion of the original image. The magnification process may degrade image characteristics such as sharpness, contrast, and / or clarity. For example, the cropped and enlarged portion of an image may have lower image characteristic scores than the original image (e.g., lower sharpness, lower contrast, and / or lower clarity). For instance, the quality degradation caused by digital zoom typically reduces edge sharpness (e.g., making edges appear blurry) and / or reduces the smoothness of curves (e.g., making curves appear to have jagged edges).

[0037] As described in more detail below, this document describes systems and techniques for determining a selected digital zoom level that preserves at least one or more image characteristics, such as sharpness, contrast, and / or clarity, at a threshold level. For example, an image sensor (e.g., a camera or part of a camera) of a device can capture images. Input identifying a portion of the image can be received at the device. For example, the device can receive input while an image is displayed on the device's screen. In some examples, the input can be touch input by touching a portion of the image displayed on the screen, such as when the screen is a touchscreen. The input can be hover input, gaze-tracking input, mouse cursor click input, pointer click input, button press input, any combination thereof, and / or another type of input identifying or selecting a portion of the image.

[0038] The device can generate multiple zoomed images from an image using digital zoom at different zoom levels (also known as zoom scales), such that each zoomed image includes a portion of the image. For example, the device can generate a first zoomed image by digitally magnifying a portion of the image using a first zoom level (e.g., 2x zoom). The device can generate a second zoomed image by digitally magnifying a portion of the image using a second zoom level greater than the first zoom level (e.g., 4x zoom). Therefore, the second zoomed image is cropped and magnified at a higher zoom level than the first zoomed image.

[0039] The device can determine the image sharpness value for each of the zoomed images. For example, the device can determine a first image sharpness for a first zoomed image and a second image sharpness for a second zoomed image. Each of the image sharpness values ​​can be, for example, a percentage of undersharpening calculated by the device using a modulation transfer function (MTF). Image sharpness measurement can be based on other measurements, such as line pairs per millimeter (mm).

[0040] A predetermined image feature threshold can be stored on the device. The device can compare different image feature scores with the image feature threshold. The zoom image with the maximum zoom intensity whose image feature score is still greater than or equal to the image feature threshold is selected as the selected zoom image. The image feature score can be an image sharpness score, and the image feature threshold can be an image sharpness threshold. In an illustrative example, the image sharpness threshold can be 80% undersharpening. In this example, if a first zoom image with 1.5x zoom has a first image sharpness score of 95% undersharpening, then a second zoom image with 2x zoom has an image sharpness score of 86% undersharpening, and a third zoom image with 2.5x zoom has an image sharpness score of 72% undersharpening, then the second zoom image (at 2x zoom) is the selected zoom image because the 86% image sharpness score of the second zoom image is greater than the 80% sharpness threshold, while the 72% image sharpness score of the third zoom image is less than the 80% sharpness threshold.

[0041] The systems and techniques described herein provide technological improvements to imaging techniques that utilize digital zoom. For example, the systems and techniques described herein maximize the intensity of digital zoom while maintaining high levels of image characteristics such as sharpness, contrast, and clarity. This provides the benefits of digital zoom, such as the ability to perform zoom operations while maintaining high image quality without the need for dedicated hardware (e.g., telephoto lenses).

[0042] Figure 1 This is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing images of a scene (e.g., an image of scene 110). The image capture and processing system 100 can capture individual images (or photographs) and / or can capture video comprising a specific sequence of multiple images (or video frames). A lens 115 of the system 100 faces scene 110 and receives light from scene 110. The lens 115 bends the light toward an image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0043] One or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from image sensor 130 and / or information from image processor 150. One or more control mechanisms 120 may include multiple mechanisms and components; for example, control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 may also include additional control mechanisms besides those shown, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0044] The focus control mechanism 125B of the control mechanism 120 can obtain the focus settings. In some examples, the focus control mechanism 125B stores the focus settings in a memory register. Based on the focus settings, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus settings, the focus control mechanism 125B can move the lens 115 closer to or further away from the image sensor 130 by actuating a motor or servo mechanism (or other lens mechanism), thereby adjusting the focus. In some cases, the system 100 may include additional lenses, such as one or more microlenses above each photodiode of the image sensor 130, each additional lens bending light received from the lens 115 toward the corresponding photodiode before reaching it. The focus settings can be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus settings can be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus settings may be referred to as image capture settings and / or image processing settings.

[0045] The exposure control mechanism 125A of the control mechanism 120 can obtain the exposure settings. In some cases, the exposure control mechanism 125A stores the exposure settings in a memory register. Based on this exposure setting, the exposure control mechanism 125A can control the aperture size (e.g., aperture size or aperture coefficient (f / stop)), the duration the aperture is open (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure settings may be referred to as image capture settings and / or image processing settings.

[0046] The zoom control mechanism 125C of the control mechanism 120 can obtain zoom settings. In some examples, the zoom control mechanism 125C stores the zoom settings in a memory register. In some examples, the zoom settings may include digital zoom settings. In some examples, the zoom settings may include optical zoom settings. Based on the zoom settings, the zoom control mechanism 125C can control which of a group of cameras is active, the focal length of a lens element assembly (lens assembly) including lens 115 and / or one or more additional lenses, or a combination thereof. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by actuating one or more motors or servo mechanisms of the zoom control mechanism 125C to move one or more of the lenses relative to each other. The zoom settings may be referred to as image capture settings and / or image processing settings.

[0047] Image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in an image generated by image sensor 130. In some cases, different photodiodes may be covered by different color filters, and thus light matching the color of the color filter covering the photodiode can be measured. For example, Bayer color filters include red, blue, and green color filters, where each pixel of the image is generated based on red light data from at least one photodiode covered by the red color filter, blue light data from at least one photodiode covered by the blue color filter, and green light data from at least one photodiode covered by the green color filter. Instead of red, blue, and / or green color filters, or other types of color filters besides red, blue, and / or green color filters, yellow, magenta, and / or cyan (also known as "emerald green") color filters may be used. Some image sensors (e.g., image sensor 130) may be completely devoid of color filters and may alternatively use different photodiodes (in some cases stacked vertically) throughout the pixel array. Different photodiodes in the pixel array can have different spectral sensitivity profiles, thus responding to light of different wavelengths. Monochrome image sensors may also lack color filters, resulting in a lack of color depth.

[0048] In some cases, image sensor 130 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which can be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output from the photodiodes and / or an analog-to-digital converter (ADC) for converting the analog signal output from the photodiodes (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more of the control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide-semiconductor (CMOS), an N-type metal-oxide-semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0049] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other type of processor 710 discussed with respect to computing system 700. Host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some embodiments, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-a-chip or SoC) including host processor 152 and ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Integrated Circuit 2 (I2C) interface, an Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a Serial General Purpose Input / Output (GPIO) interface, a Mobile Industrial Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface), an Advanced High Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one illustrative example, host processor 152 may communicate with image sensor 130 using an I2C port, and ISP 154 may communicate with image sensor 130 using a MIPI port.

[0050] Image processor 150 can perform multiple tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image identification, object identification, feature identification, receiving input, managing output, managing memory, or a combination thereof. Image processor 150 can store image frames and / or processed images in random access memory (RAM) 140 / 720, read-only memory (ROM) 145 / 725, cache memory, memory cells, another storage device, or a combination thereof.

[0051] Various input / output (I / O) devices 160 can be connected to the image processor 150. I / O devices 160 may include a display screen, keyboard, keypad, touchscreen, touchpad, touch-sensitive surface, printer, any other output device 735, any other input device 745, or some combination thereof. In some cases, subtitles can be input into the image processing device 105b via the physical keyboard or keypad of the I / O device 160 or via the virtual keyboard or keypad of the touchscreen of the I / O device 160. I / O 160 may include one or more ports, sockets, or other connectors that enable wired connections between the system 100 and one or more peripheral devices, through which the system 100 can receive data from and / or transmit data to one or more peripheral devices. I / O 160 may include one or more wireless transceivers that enable wireless connections between the system 100 and one or more peripheral devices, through which the system 100 can receive data from and / or transmit data to one or more peripheral devices. Peripheral devices may include any type of I / O device 160 previously discussed, and once they are coupled to a port, receptacle, wireless transceiver or other wired and / or wireless connector, they can be considered I / O devices 160 in themselves.

[0052] In some cases, the image capture and processing system 100 may be a single device. In other cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing system coupled to the camera). In some embodiments, the image capture device 105A and the image processing device 105B may be wirelessly coupled together, for example, via one or more wires, cables, or other electrical connectors and / or via one or more wireless transceivers. In some embodiments, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0053] like Figure 1As shown, the vertical dashed line will Figure 1 The image capture and processing system 100 is divided into two parts, namely image capture device 105A and image processing device 105B. Image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. Image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, certain components shown in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0054] Image capture and processing system 100 may include electronic devices such as mobile or landline phones (e.g., smartphones, cellular phones, etc.), desktop computers, laptops or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some implementations, image capture device 105A and image processing device 105B may be different devices. For example, image capture device 105A may include a camera device, and image processing device 105B may include a computing system, such as a mobile handheld device, desktop computer, or other computing system.

[0055] Although the image capture and processing system 100 is shown to include certain components, those skilled in the art will understand that the image capture and processing system 100 may include more than [other components]. Figure 1 The components shown are further components. Components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, components of the image capture and processing system 100 may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture and processing system 100.

[0056] In some examples, optical zoom can be controlled using lens 115, zoom control mechanism 125C, image processor 150, ISP 154, and / or main processor 152. In some examples, digital zoom can be controlled using zoom control mechanism 125C, image processor 150, ISP 154, and / or main processor 152.

[0057] Figure 2 This is a block diagram illustrating the architecture of an image processing system 200 including an image feature evaluation engine 235. The image processing system 200 includes an image sensor 205. The image sensor 205 may be an example of an image sensor 130. In some examples, the image sensor 205 may include one or more additional components (e.g., a control mechanism 120). The image sensor 205 can capture image data corresponding to an image. The image can be sent from the image sensor 205 to an image signal processor (ISP) 210, which can receive the image from the image sensor 205. The ISP 210 may be an example of an ISP 154. In some examples, the ISP 210 may include one or more additional components (e.g., an image processor 150, a host processor 152, I / O 156, I / O 160, ROM 145, RAM 140). The ISP 210 can perform some processing operations on the image to at least partially process the image. For example, the ISP 210 can perform depigmentation, color space conversion (e.g., RGB to YUV), white balance adjustment, black balance adjustment, or combinations thereof. The ISP 210 can send images to the post-processing engine 215, and the post-processing engine 215 can receive images from the ISP 210. The ISP 210 can also send images to the image feature evaluation engine 235, and the image feature evaluation engine 235 can receive images from the ISP 210.

[0058] Post-processing engine 215 may be an example of host processor 152. Post-processing engine 215 may be an example of image processor 150. In some examples, post-processing engine 215 may include one or more additional components (e.g., I / O 156, I / O 160, ROM 145, RAM 140). Post-processing engine 215 may further process the image. For example, post-processing engine 215 may perform cropping operations, resizing operations, digital zoom operations, or combinations thereof. Resizing operations may include, for example, zoom in, upsample, zoom out, downsample, subsample, rescale, resample, or combinations thereof. Resizing operations may include nearest neighbor (NN) rescaling, bilinear interpolation, bicubic interpolation, Sinc resampling, Lanczos resampling, box sampling, multilevel mipmapping, Fourier transform scaling, edge-oriented interpolation, high-quality scaling (HQX), specialized context-sensitive rescaling techniques, or combinations thereof. The post-processing engine 215 can send images to the camera preview engine 220, and the camera preview engine 220 can receive images from the post-processing engine 215. The post-processing engine 215 can also send images to the image feature evaluation engine 235, and the image feature evaluation engine 235 can receive images from the post-processing engine 215. The post-processing engine 215 can further send images to the selection interface 230, and the selection interface 230 can receive images from the post-processing engine 215. Finally, the post-processing engine 215 can send images to the media encoder 240, and the media encoder 240 can receive images from the post-processing engine 215.

[0059] The image feature evaluation engine 235 of the image processing system 200 can receive images from the ISP 210, the post-processing engine 215, the selection interface 230, the camera preview engine 220, or a combination thereof. Images received by the image feature evaluation engine 235 can be processed by the ISP 210. Images received by the image feature evaluation engine 235 can be cropped, resized, and / or otherwise processed by the post-processing engine 215. The image feature evaluation engine 235 can generate an image feature score. The image feature score can be a score indicating the sharpness, contrast, and / or clarity of the image. In some examples, the feature score can be an undersharpening percentage or an undersharpening value. In some examples, the image feature evaluation engine 235 can use a modulation transfer function (MTF) to generate the image feature score. In some examples, the feature score can be a line pair (lp) measurement per millimeter (mm) (lp / mm) value. In some examples, the image feature evaluation engine 235 can generate the image feature score based on analysis of the region of interest in the image.

[0060] Image feature evaluation engine 235 can send an image to post-processing engine 215, which can receive the image from image feature evaluation engine 235. Post-processing engine 215 can determine which scaling intensity should be used for the output image (e.g., encoded by media encoder 240) based on whether the feature score (identified by image feature evaluation engine 235) applied to each scaling intensity of the image is less than, equal to, or greater than a predetermined feature threshold. Image feature evaluation engine 235 can send an image to camera preview engine 220, which can receive the image from image feature evaluation engine 235. Camera preview engine 220 can determine which scaling intensity should be used for the preview image based on whether the feature score applied to each scaling intensity of the image is less than, equal to, or greater than a predetermined feature threshold. Image feature evaluation engine 235 can send an image to selection interface 230, which can receive the image from image feature evaluation engine 235. Camera preview engine 220 can determine which scaling intensity the selection interface 230 should use based on whether the feature score applied to each scaling intensity of the image is less than, equal to, or greater than a predetermined feature threshold.

[0061] The camera preview engine 220 of the image processing system 200 can receive images from the ISP 210, the post-processing engine 215, the selection interface 230, the image feature evaluation engine 235, or a combination thereof. The camera preview engine 220 can generate preview images based on the images it receives. The images received by the camera preview engine 220 can be processed by the ISP 210. The images received by the camera preview engine 220 can be cropped, resized, and / or otherwise processed by the post-processing engine 215. The images received by the camera preview engine 220 can be modified by the selection interface 230. The preview images can be displayed on the display screen 225 of the image processing system 200. The preview images can act as a viewfinder previewing the image data captured by the image sensor 205. The preview images can be updated periodically and / or in real time. The camera preview engine 220 is capable of sending preview images to the display screen 225, and the display screen 225 is capable of receiving images from the camera preview engine 220. The camera preview engine 220 can send preview images to the selection interface 230, and the selection interface 230 can receive images from the camera preview engine 220. The camera preview engine 220 can also send preview images to the image feature evaluation engine 235, and the image feature evaluation engine 235 can receive images from the camera preview engine 220.

[0062] The selection interface 230 of the image processing system 200 can receive images from the ISP 210, the post-processing engine 215, the camera preview engine 220, the image feature evaluation engine 235, or a combination thereof. The selection interface 230 can be associated with a touch-sensitive surface (such as a touchscreen). The display screen 225 can be a touchscreen or part of a touchscreen. The selection interface 230 can be associated with a mouse cursor, stylus, keyboard, keypad, hover input detector, gesture input detector, or another type of input device 745. The selection interface 230 can receive one or more inputs through an input interface. The input interface can correspond to I / O 156 and / or I / O 160. The selection interface 230 can be based on one or more inputs (e.g., as in...). Figure 3A A portion of an image can be identified using touch input 335. Selection interface 230 can send the identified portion of the image and / or one or more inputs themselves to post-processing engine 215, which can receive the portion of the image from selection interface 230. Post-processing engine 215 can crop, resize, and / or zoom the image based on the portion of the image identified using one or more inputs.

[0063] Selection interface 230 can be associated with a software application. For example, selection interface 230 can be associated with a face detection algorithm, a face labeling algorithm, a face tracking algorithm, an object detection algorithm, an object labeling algorithm, an object tracking algorithm, a feature detection algorithm, a feature labeling algorithm, a feature tracking algorithm, or a combination thereof. Selection interface 230 can be associated with one or more artificial intelligence (AI) algorithms, one or more trained machine learning (ML) models based on one or more ML algorithms, one or more trained neural networks (NNs), or a combination thereof. The software application associated with selection interface 230 can identify a portion of an image, for example, based on the software application detecting that a portion of the image includes a depiction of specific types of features, objects, and / or faces (e.g., such as...). Figure 3B (As shown). Selection interface 230 can send the identified portion of the image to post-processing engine 215, which can receive the portion of the image from selection interface 230. Post-processing engine 215 can crop, resize, and / or zoom the image based on the detected portion of the identified image.

[0064] In some examples, the selection interface 230 of the image processing system 200 can also be used to modify images. For example, the selection interface 230 can be associated with a software application by allowing the user to adjust image processing settings, annotate images, manually guide the post-processing engine 215 to crop images, manually guide the post-processing engine 215 to resize images, manually guide the post-processing engine 215 to zoom images, rotate images, make other changes to images, or some combination thereof. The selection interface 230 can send images and / or input and / or identified portions(s) of images to the post-processing engine 215, which can receive images from the selection interface 230. The selection interface 230 can send images and / or input and / or identified portions(s) of images to the image feature evaluation engine 235, which can receive images from the selection interface 230. The selection interface 230 can also send images and / or input and / or identified portions(s) of images to the camera preview engine 220, which can receive images from the selection interface 230.

[0065] The media encoder 240 of the image processing system 200 can receive images from the ISP 210, the post-processing engine 215, the camera preview engine 220, the image feature evaluation engine 235, the selection interface 230, or a combination thereof. Images received by the camera preview engine 220 can be processed by the ISP 210. Images received by the camera preview engine 220 can be cropped, resized, and / or otherwise processed by the post-processing engine 215. Images received by the camera preview engine 220 can be modified by the selection interface 230. In some examples, the post-processing engine 215 can generate a magnified version of the image based on a feature score selected by the image feature evaluation engine 235 for different scaling intensities. The post-processing engine 215 can send the magnified version of the image to the media encoder 240, which can receive the magnified version of the image from the post-processing engine 215. The media encoder 240 can encode one or more images received by the media encoder 240 using still image encoding techniques, such as still image compression techniques. Media encoder 240 may encode one or more images received by media encoder 240 using video encoding techniques, such as video compression techniques. Video encoding techniques may use images as one of multiple video frames in a video and may involve inter-frame decoding and / or intra-frame decoding. Media encoder 240 may generate encoded images. These encoded images may be referred to as output images. In some cases (e.g., from processing engine 215 and / or any other transmitting component discussed herein), images sent to media encoder 240 may be referred to as output images. Media encoder 240 may send images to camera preview engine 220, which may receive images from media encoder 240. Media encoder 240 may send images to display screen 225, which may receive images from media encoder 240. Display screen 225 may display the images it received from media encoder 240.

[0066] The display screen 225 of the image processing system 200 can receive images from the media encoder 240, the ISP 210, the post-processing engine 215, the camera preview engine 220, the image feature evaluation engine 235, the selection interface 230, or a combination thereof. The display screen 225 can display the images. The display screen 225 can be any type of display screen or other display device (e.g., a projector). The display screen 225 can be or may include any element discussed with respect to the output device 735. For example, the display screen 225 may also include a speaker that can output audio corresponding to the image (e.g., if the image is a video frame of a video including an audio track). The display screen 225 may correspond to I / O 156 and / or I / O 160.

[0067] Figure 3AThis is a conceptual diagram 300A showing a selection interface, where a digitally zoomed portion 320 of image 310 centers on a portion 330 of image 310 identified by touch input 335. Image 310 shows an office environment with four people sitting or standing around a set of tables and chairs. A fingerprint icon is overlaid on... Figure 3A The fingerprint icon represents touch input 335 received via a selection interface (e.g., selection interface 230). The selection interface may include a touch-sensitive surface, such as the touch-sensitive surface of a touchscreen. The selection interface identifies a designated portion 330 of image 310 based on the position of touch input 335 on the display screen and based on the positioning of image 310 displayed on the display screen. The designated portion 330 includes at least a subset of the area covered by touch input 335. The designated portion 330 of image 310 is shown as a circular portion 330 of image 310, outlined using dashed lines. In some cases, the designated portion 330 of image 310 may simply be a point representing the center of the area covered by touch input 335, or a cursor click, or a point touched by a stylus, or some combination thereof.

[0068] The selection interface identifies the digitally scaled portion 320 of image 310 based on the identified portion 330 of image 310. For example, Figure 3A The digitally scaled portion 320 of image 310 is centered on the identified portion 330 of image 310. The digitally scaled portion 320 of image 310... Figure 3A The image is shown as a shaded rectangle outlined by dashed lines. The digital zoom portion 320 of image 310 represents the cropped area of ​​image 310 used for digital zoom operations. For example, in a digital zoom operation, everything outside the digital zoom portion 320 of image 310 is cropped, leaving only the digital zoom portion 320 of image 310. The digital zoom operation can adjust the size of the digital zoom portion 320 of image 310 before or after cropping (e.g., zoom in and / or upsample).

[0069] Figure 3B The conceptual diagram 300B shows the selection interface, in which the digitally scaled portion 340 of the image 310 is centered on the portion 350 of the image 310 based on image analysis identification. Figure 3B Image 310 shown is Figure 3AThe same image 310 of the office environment shown. Image analysis identifies a person's face in image 310, as represented by a dashed rounded rectangle surrounding the face, and is identified as the identified portion 350 of image 310. In some examples, image analysis identifies portion 350 of image 310 by detecting faces using face detection algorithm 355. In some examples, image analysis identifies portion 350 of image 310 by identifying faces using face identification algorithm 355. The identified portion 350 of image 310 may be a bounding box around the face generated by face-based detection image analysis. A digitally scaled portion 340 of image 310 is based on the identified portion 350 of image 310. For example, Figure 3B The digitally scaled portion 340 of image 310 is centered on the identified portion 350 of image 310. The digitally scaled portion 340 of image 310 represents the cropped area of ​​image 310 used for digital zoom operations. Figure 3B The rectangle shown is a shaded rectangle outlined by dashed lines.

[0070] Although the digitally scaled portion 340 of image 310 is shown centered on the identified portion 350 of image 310, this is not necessary. In some examples, the digitally scaled portion 340 of image 310 may include the identified portion 350 of image 310 without being centered on the identified portion 350 of image 310. In some examples, the digitally scaled portion 340 of image 310 may be identified based on the identified portion 350 of image 310, without including the identified portion 350 of image 310. For example, the identified portion 350 of image 310 may represent a corner or side of the digitally scaled portion 340 of image 310. Similarly, although the digitally scaled portion 340 of image 310 is shown centered on the identified portion 330 of image 310, this is not necessary. In some instances, the digitally scaled portion 340 of image 310 may include the identified portion 330 of image 310 without being centered on the identified portion 330 of image 310. In some instances, the digitally scaled portion 340 of image 310 may be identified based on the identified portion 330 of image 310, without including the identified portion 330 of image 310. For example, the identified portion 330 of image 310 may represent a corner or side of the digitally scaled portion 340 of image 310.

[0071] In some examples, instead of or in addition to face identification algorithm 355, a semantic analysis algorithm that performs semantic analysis on image 310 may be used. Semantic analysis may, for example, perform segmentation between one or more foreground elements and one or more background elements. Semantic analysis may be performed based on analysis of the image itself, analysis of other images of the same scene (e.g., previous image frames in a video of the scene), analysis of depth information from depth sensors (e.g., from LIDAR, RADAR, SONAR, SODAR, time-of-flight sensors, structured light sensors, or combinations thereof), or combinations thereof. For example, an identified portion 350 of image 310 may be selected based on semantic analysis of image 310, which indicates that the identified portion 350 includes depictions of foreground elements. In some examples, an identified portion of image 310 may be selected based on semantic analysis of image 310, which indicates that the identified portion 350 includes depictions of background elements. Semantic analysis may identify topics of interest. For example, the identified portion 350 of image 310 can be selected based on semantic analysis of image 310, which indicates that the identified portion 350 includes a depiction of a subject of interest. Semantic analysis can identify regions of interest. For example, the identified portion 350 of image 310 can be selected based on semantic analysis of image 310, which indicates that the identified portion 350 includes a region of interest. Semantic analysis can identify the scene depicted in the image. For example, the identified portion 350 of image 310 can be selected based on semantic analysis of image 310, which indicates that the identified portion 350 includes a depiction of a specific portion of the scene depicted in the image.

[0072] In some examples, it can be based on one or more inputs from the input interface (e.g., Figure 3A Touch input 335) and data from image analysis (e.g., Figure 3B The digitally scaled portion of an image is identified by a combination of one or more inputs from a face detection algorithm 355, an object detection algorithm, a feature detection algorithm, semantic analysis discussed above, or a combination thereof. For example, image analysis can identify the location of the input from the input interface near the facial depiction in image 310. The selected interface can center the digitally scaled portion of the image at a point along a line between the center of the input from the input interface and the center of the bounding box including the detected facial depiction in image 310. For example, the point could be at the midpoint along this line.

[0073] Figure 4AThis is a conceptual diagram 400A showing image 410 and a corresponding sharpness curve diagram 415. Image 410 depicts a bird. One side of the bird's head is depicted in image 410. One eye of the bird is depicted in image 410. A square is overlaid on image 410 around the bird's eye and represents the identified portion 450 of image 410. The identified portion 450 of image 410 can be identified by a selection interface based on input from an input interface (e.g., touch input above the bird's eye), based on image analysis (e.g., an object detector identifying the bird's eye), or based on a combination thereof.

[0074] exist Figures 4A-4C In this context, the sharpness score is used as an image feature score. Specifically, in... Figures 4A-4C In this diagram, the undersharpness value is used as an image feature score. Image 410 has a high image feature score of 100% (high undersharpness value 455). Sharpness curve 415 shows the sharpness score of image 410. Sharpness curve 415 includes a vertical axis along which the results of the modulation transfer function (MTF) are plotted. The results of the MTF can be referred to as modulation, feature score, sharpness, percentage undersharpness, contrast, spatial frequency response, or some combination thereof. In some examples, the MTF can be identified based on the equation modulation = (Imax - Imin) / (Imax + Imin). In this equation, Imax can represent the maximum intensity, and Imin can represent the minimum intensity. Sharpness curve 415 includes a horizontal axis along which the period per pixel is plotted. The period per pixel can indicate how many periods of alternation between maximum intensity lines (e.g., black lines) and minimum intensity lines (e.g., white lines) per pixel are (or will be) discernible at the current sharpness level (e.g., feature score).

[0075] Figure 4B It is shown Figure 4A A conceptual diagram 400B shows the first digitally scaled portion 420 of image 410 and the corresponding sharpness curve 425. The first digitally scaled portion 420 of image 410 is based on a first scaling intensity. Figure 4A The image 410 is a cropped and resized portion (e.g., enlarged and / or upsampled). The first digitally scaled portion 420 of image 410 is centered on the identified portion 450 of the image. The first digitally scaled portion 420 of image 410 has a high feature score of 100% - a high undersharpening value 460. Sharpness curve 425 shows the sharpness score of image 410.

[0076] Figure 4C It is shown Figure 4A A conceptual diagram 400C shows the second digital scaling portion 430 of image 410 and the corresponding sharpness curve 435. In some examples, the second digital scaling portion 430 of image 410 is based on a ratio... Figure 4B The first scaling strength used is stronger than the second scaling strength. Figure 4A The cropped and resized portion (e.g., enlarged and / or upsampled) of image 410. In some examples, the second digitally scaled portion 430 of image 410 is... Figure 4B The portion of the first digitally scaled portion 420 of the image 410 that has been cropped and resized (e.g., enlarged and / or upsampled).

[0077] The second digitally scaled portion 430 of image 410 includes the identified portion 450 of the image. The second digitally scaled portion 430 of image 410 is vertically centered on the identified portion 450 of the image. The second digitally scaled portion 430 of image 410 is not horizontally centered on the identified portion 450 of the image, wherein the identified portion 450 of the image is slightly to the left of the center of the second digitally scaled portion 430 of image 410. The second digitally scaled portion 430 of image 410 has a lower feature score of 72% – a lower undersharpening value of 465. Sharpness curve 435 shows the sharpness score of image 410.

[0078] To maintain image sharpness, the image processing device can output a version of image 410 with the highest scaling intensity (the most magnified version of image 410), which still has a feature score above a predetermined feature threshold. Here, Figure 4A The feature score of image 410 is 100%. Figure 4B The feature score of the first digital scaling portion 420 of image 410 is 100%, and Figure 4C The characteristic score of the second digitally scaled portion 430 of image 410 is 72%. If a predetermined characteristic threshold is higher than 72% (e.g., 75%, 80%, 85%, 90%, 95%, 99%), then the image processing device outputs... Figure 4B The first digitally scaled portion 420 of image 410 is considered because its 100% feature score exceeds a predetermined feature threshold. If the predetermined feature threshold is below 72% (e.g., 70%, 65%, 60%, 55%, 50%), then... Figure 4C The second digitally scaled portion 430 of the image 410 can be output by the image processing device because its 72% feature score exceeds a predetermined feature threshold.

[0079] exist Figures 4A to 4C In the context of the image 410, a feature score is generated based on sharpness analysis of the identified portion 450 (e.g., as discussed with respect to image feature evaluation engine 235). The identified portion 450 of the image 410 includes... Figure 4A Image 410 Figure 4B The first digital scaling portion 420 of image 410 and Figure 4CIn the second digitally scaled portion 430 of image 410. Therefore, a characteristic score based on the identified portion 450 of image 410 as depicted in each of these can provide Figure 4A Image 410 Figure 4B The first digital scaling portion 420 of image 410 and Figure 4C A direct comparison between the second digitally scaled portion 430 of image 410.

[0080] In some examples, it can be based on Figure 4A The sharpness analysis of the entire image 410 is used to generate Figure 4A The feature score of image 410. In some examples, it can be based on Figure 4B Image 410 is generated by sharpness analysis of the entire first digitally scaled portion 420. Figure 4B The feature score of the first digital scaling portion 420 of image 410. In some examples, it can be based on... Figure 4C The sharpness analysis of the entire second digital scaling portion 430 of image 410 is used to generate the image. Figure 4C The characteristic score of the second digital scaling portion 430 of image 410.

[0081] Figure 5 This is a conceptual diagram 500 illustrating different zoom intensities and corresponding feature scores along a series of incremental zoom operations. An image processing system receives an image captured by an image sensor. A first zoom intensity 510 of 1X corresponds to the image received from the image sensor. The image processing system performs sharpness analysis on the image at the first zoom intensity 510 of 1X and generates a first feature score 515 that is “very high”.

[0082] The image processing system performs a digital magnification operation on the original image to produce a zoomed image with a second zoom intensity of 520 (2X). The image processing system performs a sharpness analysis on the zoomed image with the second zoom intensity of 520 (2X) and generates a "high" second characteristic score of 525.

[0083] In some examples, the image processing system performs a second digital magnification operation to increase the zoom image by an additional factor of 2X with a second scaling intensity 520 of 2X, thereby producing a zoom image with a total third scaling intensity 530 of 4X. In some examples, the image processing system can directly generate a zoom image with a third scaling intensity 530 of 4X from the original image by directly applying the third scaling intensity 530 of 4X to the original image. The image processing system performs a sharpness analysis on the zoom image with a third scaling intensity 530 of 4X and generates a "medium" third characteristic score 535.

[0084] In some examples, the image processing system performs a third digital magnification operation to upscale the zoomed image by an additional factor of 2X with a third scaling intensity of 4X 530, producing a zoomed image with a total fourth scaling intensity of 8X 540. In some examples, the image processing system can directly generate an 8X zoomed image with a fourth scaling intensity of 540 from the original image by applying the total fourth scaling intensity of 8X directly to the original image. The image processing system performs a sharpness analysis on the zoomed image with the fourth scaling intensity of 540 and generates a "low" fourth characteristic score of 545.

[0085] In some examples, the image processing system performs a fourth digital magnification operation to upscale the zoomed image by an additional factor of 2X with a fourth scaling intensity of 8X 540, producing a zoomed image with a total fifth scaling intensity of 16X 550. In some examples, the image processing system can directly generate a zoomed image with a fifth scaling intensity of 550 from the original image by applying the 16X fifth scaling intensity of 550 directly to the original image. The image processing system performs a sharpness analysis on the zoomed image with the fifth scaling intensity of 550 16X and generates a fifth feature score of "very low" 555.

[0086] In some examples, the image processing system identifies a fourth feature score 545 as less than a predetermined feature threshold 560. For example, the predetermined feature threshold 560 could be "Medium," and the third feature score 535 is "Medium," thus equal to the predetermined feature threshold 560. In some examples, the image processing system may determine that it will output a zoom image with a third zoom intensity 530 of 4X based on the third feature score 535 ("Medium") meeting or exceeding the predetermined feature threshold 560 ("Medium"). In some examples, meeting the predetermined feature threshold 560 may not be sufficient, and the image processing system may require a feature score exceeding the predetermined feature threshold 560. In such examples, the image processing system may determine that it will output a zoom image with a second zoom intensity 520 of 2X based on the second feature score 525 ("High") meeting or exceeding the predetermined feature threshold 560 ("Medium").

[0087] Figure 6This is a flowchart 600 illustrating operations for processing image data. In some examples, the operations of the image processing techniques shown in flowchart 600 can be performed by an image processing system. In some examples, the image processing system is image processing system 200. In some examples, the classification system includes image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, image processing system 200, image sensor 205, ISP 210, post-processing engine 215, selection interface 230, image feature evaluation engine 235, media encoder 240, and... Figures 3A-3B Related selection interfaces, and Figures 4A-4C The associated image processing system, the trained ML model, the trained NN, one or more web servers for cloud services, computing system 700, or a combination thereof, at least one of these.

[0088] At operation 605, the image processing system receives an image from the image sensor. The image can be captured by the image sensor of the image processing system. Examples of image sensors for operation 605 include... Figure 1 Image sensor 130 and Figure 2 Image sensor 205. In some examples, the image is processed at least partially (e.g., by ISP 154 and / or ISP 210) before the image processing system receives the image. In some examples, the image processing system may include a connector coupled to the image sensor, and the connector may be used to receive the image. The connector may include a port, socket, wire, input / output (IO) pin, conductive trace on a printed circuit board (PCB), any other type of connector discussed herein, or some combination thereof. In some examples, the image processing system may include an image sensor that captures images.

[0089] At operation 610, the image processing system determines a first image feature score for the first image dataset. The first image dataset corresponds to a first scaling variant of the image at a first scaling intensity. In some examples, the image processing system generates the first scaling variant of the image at least in part by cropping and enlarging the image according to the first scaling intensity. Figure 4B The first digital scaling portion 420 of image 410 is an example of a first scaling variant (and / or a first image data set) of the image at a first scaling intensity.

[0090] At operation 615, the image processing system determines a second image feature score for the second image dataset. The second image dataset corresponds to a second scaling variant of the image at a second scaling intensity. In some examples, the image processing system generates the second scaling variant of the image at least in part by cropping and enlarging the image according to the second scaling intensity and at least one of the first scaling variant of the image. Figure 4C The second digital scaling portion 430 of image 410 is an example of a second scaling variant (and / or a second image data set) of the image under the first scaling intensity.

[0091] In some examples, the first image dataset corresponds to a first digitally scaled portion of an image with a first scaling intensity. Figure 4B The first digitally scaled portion 420 of image 410 is an example of a first digitally scaled portion of an image at a first scaling intensity (and / or a first image data set). In some examples, a second image data set corresponds to a second digitally scaled portion of an image at a second scaling intensity. Figure 4C The second digitally scaled portion 430 of image 410 is an example of a second digitally scaled portion (and / or a second image data set) of the image under the first scaling intensity. In some examples, the second digitally scaled portion of the image is a subset of the first digitally scaled portion of the image. For example, Figure 4C The second digital scaling part 430 is Figure 4B The first digitally scaled portion 420 of image 410 is a subset of the second digitally scaled portion of the image. In some examples, the first digitally scaled portion of the image is different from the second digitally scaled portion of the image. For example, Figure 4C The second digital scaling part 430 is different Figure 4B The first digital scaling portion 420 of image 410.

[0092] In some examples, a first digitally scaled portion of an image generated with a first scaling intensity may include cropping the image. In some examples, a second digitally scaled portion of an image generated with a second scaling intensity may include cropping the image or the first digitally scaled portion of the image. In some examples, a first digitally scaled portion of an image generated with a first scaling intensity may include enlarging and / or upsampling the image. In some examples, a second digitally scaled portion of an image generated with a second scaling intensity may include enlarging and / or upsampling the first digitally scaled portion of the image.

[0093] In some examples, the image sensor may be a high-resolution image sensor. In operation 605, the image may be pixel-binned as received from the image sensor. In pixel binning, the values ​​(e.g., charge values) of one or more neighboring pixels (e.g., the square of four pixels) may be averaged or summed and reported as a single superpixel. Pixel binning reduces noise. In some examples, the image processing system may disable binning and may use the raw pixel values ​​from the image sensor for the zoom images(s) instead of superpixels, instead of magnifying and / or upsampling the image to generate a first digitally scaled portion and / or a second digitally scaled portion of the image. In some examples, instead of magnifying and / or upsampling the image to generate a first digitally scaled portion and / or a second digitally scaled portion of the image, the image processing system may reduce binning by combining fewer raw pixels for each superpixel, and may reduce binned superpixel values ​​for the zoom images(s).

[0094] At operation 620, the image processing system identifies that the second image feature score is less than the image feature threshold. In some examples, the image processing system also identifies (operation 610) that the first image feature score is greater than or equal to the image feature threshold.

[0095] In some examples, the first image feature score is a first image sharpness score, the second image feature score is a second image sharpness score, and the image feature threshold is an image sharpness threshold. In some examples, the first image feature score is a first undersharpened value of a first image dataset, the second image feature score is a second undersharpened value of a second image dataset, and the image feature threshold is an undersharpened value threshold.

[0096] In the illustrative example, the image feature threshold for operation 615 can be: Figure 5 The image feature threshold is 560. The second image feature score of operations 615 and 620 can be... Figure 5 The image feature score is 545 (low) or 555 (very low). Therefore, the second scaling variant of the image at the second scaling intensity can be... Figure 5 The scaling variation of the image at a scaling intensity of 540 (8X) or a scaling intensity of 550 (16X). The first image feature score of operation 610 can be Figure 5 The image feature score is 515 (very high), 525 (high), or 535 (medium). Therefore, the first scaling variation of the image at the first scaling intensity can be... Figure 5 Scaling variations of images at scaling intensities of 510 (1X), 520 (2X), or 530 (4X).

[0097] At operation 625, the image processing system outputs the first image dataset as the output image based on the fact that the score of the second image feature is less than the image feature threshold. In some instances, outputting the first image dataset as the output image is also based on the fact that the score of the first image feature is greater than or equal to the image feature threshold.

[0098] In some instances, the first scaling variant of the first image dataset and / or the output image and / or the image at the first scaling intensity can be Figure 5 Scaling variations of the image at scaling intensities of 510 (1X), 520 (2X), or 530 (4X). In an illustrative example, the first scaling variation of the first image dataset and / or the output image and / or the image at the first scaling intensity can be... Figure 5 The scaling variation of the image at scaling intensity 530 (4X) is because scaling intensity 530 (4X) corresponds to an image feature score ("medium" image feature score 535) that is not less than... Figure 5 The highest possible scaling intensity is the image characteristic threshold of 560 (“medium”).

[0099] In some examples, to output a first image dataset as an output image, the image processing system is configured and can render the output image for display on a display screen. In some examples, to output a first image dataset as an output image, the image processing system is configured and can be used to display the output image on a display screen. The image processing system may include a display screen. Examples of a display screen may include I / O 156, I / O 160, display screen 225, output device 735, or a combination thereof. In some examples, to output a first image dataset as an output image, the image processing system is configured and can encode the output image for display on a display screen. In some examples, the encoding and / or rendering of the output image may be performed by a host processor 152, an image processor 150, a post-processing engine 215, a camera preview engine 220, a media encoder 240, output device 735, or a combination thereof.

[0100] In some examples, to output a first image dataset as an output image, one or more processors are configured to send the output image using a communication interface. The image processing system may include a communication interface that can be configured to send the output image to a device. Examples of communication interfaces may include I / O 156, I / O 160, output device 735, communication interface 740, or combinations thereof.

[0101] In some examples, the image processing system also receives one or more inputs identifying a portion of an image, wherein a first scaling variant and a second scaling variant of the image are based on the identified portion of the image. Examples of identified portions of the image include... Figure 3A The portion 330 identified in image 310 Figure 3A The digital scaling portion of image 310, 320, Figure 3B The portion 350 identified in image 310 Figure 3B The digital scaling portion of image 310 is 340. Figures 4A-4C The portion 450 identified by image 410 Figure 4B The first digital scaling portion 420 of image 410 Figure 4C The second digital scaling portion 430 of the image 410 or a combination thereof.

[0102] In some examples, the first scaling variant and the second scaling variant of the image include the identified portion of the image. In some examples, one or more inputs include user input via a user interface. In some examples, the user input and / or one or more inputs include at least one of touch input, hover input, gesture input, gaze input, button press, pointer movement, pointer click, or a combination thereof. Examples of user input and / or one or more inputs include... Figure 3A Touch input 335. In some examples, the image processing system receives one or more inputs from an object detection algorithm, which determines that the identified portions of the image include a depiction of the type of object. In some examples, the object type is a face. Examples of object detection algorithms include... Figure 3B The face detection algorithm is 355. The object detection algorithm can be a face detection algorithm, a face labeling algorithm, a face tracking algorithm, an object detection algorithm, an object labeling algorithm, an object tracking algorithm, a feature detection algorithm, a feature labeling algorithm, a feature tracking algorithm, a semantic analysis algorithm, or a combination thereof.

[0103] In some examples, the image processing system may receive a first sensor readout corresponding to a first portion of the scene. The image processing system may evaluate the readout. The image processing system may receive a second sensor readout corresponding to a second portion of the scene at different scaling intensities, and the image processing system may select either the first or second sensor readout and output it as an output image. In some examples, the image processing system may determine a first image feature score for the first sensor readout and a second image feature score for the second sensor readout. The image processing system may, for example, select the first sensor readout as the output image based on determining that the first image feature score is higher than the second image feature score. The image processing system may, for example, select the first sensor readout as the output image based on determining that the first image feature score is higher than an image feature threshold and / or the second image feature score is lower than an image feature threshold.

[0104] The image processing techniques shown in flowchart 600 may also include any operations shown in or discussed in any of the concept diagrams, block diagrams and flowcharts 100, 200, 300, 500 and / or 700.

[0105] In some cases, at least a subset of the techniques illustrated in any of the concept diagrams, block diagrams, and flowcharts 100, 200, 300, 500, and / or 700 can be remotely executed by one or more web servers of a cloud service. In some examples, the processes described herein (e.g., including those shown by concept diagrams, block diagrams, and flowcharts 200, 300, 500, 600, 1000, 1100, 1200, 600, and / or other processes described herein) can be executed by a computing system or apparatus. In some instances, the processes illustrated in concept diagrams, block diagrams, and flowcharts 100, 200, 300, 500, and / or 700 can be executed by... Figure 1 Image capture device 105A, Figure 1 Image processing device 105B and / or Figure 1 The image capture and processing system 100 performs the operation. In some examples, the processes shown in the concept diagrams, block diagrams, and flowcharts 100, 200, 300, 500, and / or 700 can be performed by... Figure 2 The image processing system 200 performs the operation. In some examples, the processes shown in the concept diagrams, block diagrams, and flowcharts 100, 200, 300, 500, and / or 700 can be performed by a system with... Figure 7 The computing system 700 shown is an architecture for computing system execution. The computing system may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device for an autonomous vehicle, a robotic device, a television, and / or any other computing device with the resource capability to perform the processes described herein (including the processes shown in concept diagrams, block diagrams, and flowcharts 100, 200, 300, and 500). In some cases, the computing system or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or multiple other components configured to perform the steps of the processes described herein. In some examples, the computing system may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive Internet Protocol (IP) based data or other types of data.

[0106] Components of a computing system can be implemented in circuits. For example, components may include electronic circuits or other electronic hardware and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0107] The processes illustrated in concept diagrams, block diagrams, and flowcharts 100, 200, 300, 500, and / or 700 are organized as logical flowcharts, whose operations represent a series of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media, which, when executed by one or more processors, performs the operation. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0108] Furthermore, the processes illustrated in concept diagrams, block diagrams, and flowcharts 100, 200, 300, 500, and 700, and / or other processes described herein, can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, via hardware, or a combination thereof. As described above, the code can be stored, for example, on a computer-readable or machine-readable storage medium in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0109] Figure 7 This is a diagram illustrating an example of a system used to implement certain aspects of this technology. Specifically, Figure 7 An example of a computing system 700 is shown, which can be any computing device or system comprising, for example, the following: an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an image processor 150, a host processor 152, an ISP 154, an image processing system 200, an image sensor 205, an ISP 210, a post-processing engine 215, a selection interface 230, an image feature evaluation engine 235, a media encoder 240, and... Figures 3A-3BRelated selection interfaces, and Figures 4A-4C An associated image processing system, one or more trained ML models, one or more trained neural networks, one or more web servers, a camera, any combination thereof, or any component thereof, wherein the components of the system communicate with each other using connection 705. Connection 705 may be a physical connection using a bus, or a direct connection to processor 710, such as in a chipset architecture. Connection 705 may also be a virtual connection, a networking connection, or a logical connection.

[0110] In some embodiments, the computing system 700 is a distributed system, wherein the functions described herein may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a plurality of such components, each performing some or all of the functions described for that component. In some embodiments, a component may be a physical or virtual device.

[0111] Example system 700 includes at least one processing unit (CPU or processor) 710 and a connection 705 that couples various system components, including system memory 715 (such as read-only memory (ROM) 720 and random access memory (RAM) 725), to processor 710. Computing system 700 may include a cache 712 of high-speed memory that is directly connected to, adjacent to, or integrated into processor 710.

[0112] Processor 710 may include any general-purpose processor and hardware or software services, such as services 732, 734, and 736 stored in storage device 730, which are configured to control processor 710 and dedicated processors, where software instructions are incorporated into the actual processor design. Processor 710 can essentially be a completely independent computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0113] To enable user interaction, the computing system 700 includes an input device 745, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. The computing system 700 may also include an output device 735, which can be one or more of multiple output mechanisms. In some examples, a multimodal system allows the user to provide multiple types of input / output to communicate with the computing system 700. The computing system 700 may include a communication interface 740, which typically controls and manages user input and system output. The communication interface can use wired and / or wireless transceivers to perform or facilitate the reception and / or transmission of wired or wireless communications, including utilizing audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple® Lightning® ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs, Bluetooth® wireless signal transmission, Bluetooth® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communication (DSRC) wireless signal transmission, and 802.11. The communication interface 740 may include Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), global microwave access interoperability (WiMAX), infrared (IR) wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 740 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing system 700 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There are no restrictions on operation on any particular hardware arrangement, and therefore the basic features here can be easily replaced with improved hardware or firmware arrangements as they are developed.

[0114] Storage device 730 may be a non-volatile and / or non-transitory and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as magnetic tape cassettes, flash memory cards, solid-state storage devices, digital multifunction disks, cassette tapes, floppy disks, flexible disks, hard disks, magnetic tapes, magnetic stripes, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, optical disc read-only memory (CD-ROM), optical disc rewritable optical disc (CD), optical disc rewritable optical disc (CD). Digital Video Disc (DVD), Blu-ray Disc (BDD), Holographic Disc, Another Optical Medium, Secure Digital (SD) Card, Micro Secure Digital (microSD) Card, Card, Smart Card Chip, EMV Chip, Subscriber Identity Module (SIM) Card, Mini / Micro / Nano / PicoSIM Card, Another Integrated Circuit (IC) Chip / Card, Random Access Memory (RAM), Static RAM (SRAM), Dynamic RAM (DRAM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash EPROM, Cache Memory (L1 / L2 / L3 / L4 / L5 / L#), Resistive Random Access Memory (RRAM / ReRAM), Phase Change Memory (PCM), Spin-Torque RAM (STT-RAM), Another Memory Chip or Container, and / or combinations thereof.

[0115] Storage device 730 may include software services, servers, etc., which enable the system to perform functions when the code defining such software is executed by processor 710. In some embodiments, hardware services that perform a particular function may include software components stored in a computer-readable medium and associated with necessary hardware components, such as processor 710, connection 705, output device 735, etc., to perform that function.

[0116] As used herein, the term "computer-readable medium" includes (but is not limited to) portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data can be stored but do not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include (but are not limited to) magnetic disks or magnetic tapes, optical storage media (e.g., optical discs (CDs) or digital versatile optical discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may have code and / or machine-executable instructions stored thereon, which may represent programs, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments can be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted using any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0117] In some embodiments, computer-readable storage devices, media, and memories may include cable or wireless signals, including bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0118] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that embodiments can be practiced without these specific details. For clarity, in some instances, the technology may be presented as comprising individual functional blocks, including functional blocks comprising devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the accompanying drawings and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0119] The various embodiments described above can be illustrated as processes or methods depicted as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0120] The processes and methods described in the examples above can be implemented using computer-executable instructions stored in or otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a particular function or group of functions. Parts of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include hard disks or optical disks, flash memory, USB devices equipped with non-volatile memory, networked storage devices, etc.

[0121] Devices implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. One or more processors may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mount devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. As another example, such functionality may also be implemented on a circuit board between different chips or different processes that execute in a single device.

[0122] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0123] In the foregoing description, various aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, while illustrative embodiments of this application have been described in detail herein, it should be understood that the inventive concept can be implemented and employed in other different ways, and the appended claims are intended to be construed as including such variations, in addition to being limited by the prior art. Various features and aspects of the above-described applications can be used individually or in combination. Furthermore, embodiments can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in a different order than that described.

[0124] Those skilled in the art will understand that, without departing from the scope of this specification, the less than ("<") and greater than (">") symbols or terms used herein may be replaced by the less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively.

[0125] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0126] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0127] The use of claim language or other languages ​​to state "at least one" and / or "one or more" in a set indicates that one or more members of the set (in any combination) satisfy the claim. For example, claim language stating "at least one of A and B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The use of "at least one" and / or "one or more" in the language set does not limit the set to items listed in the set. For example, claim language stating "at least one of A and B" can mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0128] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above in general terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0129] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication handsets, or integrated circuit devices, having a wide range of uses including applications in wireless communication handsets and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as a discrete but interoperable logic device. If implemented in software, the techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (e.g., synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. Alternatively or concurrently, the technology may be implemented at least in part by a computer-readable communication medium carrying or conveying program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagating signals or waves.

[0130] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. This processor can be configured to perform any of the techniques described herein. A general-purpose processor may be a microprocessor; however, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, a combination of one or more microprocessors with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within a dedicated software or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0131] The illustrative aspects of this disclosure include:

[0132] Aspect 1: An apparatus for processing image data, the apparatus comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: receive an image from an image sensor; determine a first image feature score of a first image data set, the first image data set corresponding to a first scaling variation of the image at a first scaling intensity; determine a second image feature score of a second image data set, the second image data set corresponding to a second scaling variation of the image at a second scaling intensity; identify that the second image feature score is less than an image feature threshold; and output the first image data set as an output image based on the identification that the second image feature score is less than the image feature threshold.

[0133] Aspect 2: According to the apparatus of aspect 1, wherein outputting the first image dataset is based on the output image being greater than or equal to the image feature threshold.

[0134] Aspect 3: The apparatus according to any one of Aspects 1 to 2, wherein the first image feature score is a first image sharpness score, the second image feature score is a second image sharpness score, and the image feature threshold is an image sharpness threshold.

[0135] Aspect 4: The apparatus according to any one of aspects 1 to 3, wherein the first image feature score is a first undersharpened value of the first image dataset, the second image feature score is a second undersharpened value of the second image dataset, and the image feature threshold is an undersharpened value threshold.

[0136] Aspect 5: An apparatus according to any one of aspects 1 to 4, wherein the one or more processors are configured to: receive one or more inputs identifying a portion of the image, wherein the first scaling variant of the image and the second scaling variant of the image are based on the identified portion of the image.

[0137] Aspect 6: The apparatus according to aspect 5, wherein the first scaling variant of the image and the second scaling variant of the image include the identified portion of the image.

[0138] Aspect 7: The apparatus according to any one of aspects 5 or 6, wherein one or more inputs include user input via a user interface.

[0139] Aspect 8: The apparatus according to any one of aspects 5 to 7, wherein one or more inputs include at least one of touch input, gesture input and gaze input.

[0140] Aspect 9: The apparatus according to any one of aspects 5 to 8, wherein the one or more processors are configured to receive the one or more inputs from an object detection algorithm, the object detection algorithm determining that the identified portion of the image includes a depiction of the type of object.

[0141] Aspect 10: The apparatus according to aspect 9, wherein the type of object is a face.

[0142] Aspect 11: The apparatus according to any one of aspects 1 to 10, wherein one or more processors are configured to: generate a first scaling variant of an image by cropping and enlarging the image according to a first scaling intensity at least in part; and generate a second scaling variant of an image by cropping and enlarging the image according to a second scaling intensity and at least one of the first scaling variant of the image.

[0143] Aspect 12: The apparatus according to any one of aspects 1 to 11, wherein, in order to output a first image dataset as an output image, one or more processors are configured to render the output image for display on a display screen.

[0144] Aspect 13: The apparatus according to any one of aspects 1 to 12 further includes: a display screen configured to display an output image.

[0145] Aspect 14: The apparatus according to any one of Aspects 1 to 13, wherein one or more processors are configured to: identify that the difference between a first image feature score and a second image feature score is less than a difference threshold; and output the second image dataset as a second output image based on the difference being less than the difference threshold.

[0146] Aspect 15: The apparatus according to any one of aspects 1 to 14, wherein the first image data set corresponds to a first digitally scaled portion of the image under a first scaling intensity, and the second image data set corresponds to a second digitally scaled portion of the image under a second scaling intensity.

[0147] Aspect 16: The apparatus according to aspect 15, wherein the second digitally scaled portion of the image is a subset of the first digitally scaled portion of the image.

[0148] Aspect 17: The apparatus according to aspect 15, wherein a first digitally scaled portion of an image is different from a second digitally scaled portion of an image.

[0149] Aspect 18: The apparatus according to any one of aspects 1 to 17, wherein, in order to output a first image dataset as an output image, one or more processors are configured to send the output image using a communication interface.

[0150] Aspect 19: The apparatus according to any one of aspects 1 to 18 further includes: a communication interface configured to send an output image to the device.

[0151] Aspect 20: The apparatus according to any one of aspects 1 to 19, wherein the apparatus is one of a mobile device, a mobile handheld device, a wireless communication device, a head-mounted display, and a camera.

[0152] Aspect 21: The apparatus according to any one of aspects 1 to 20 further includes: an image sensor configured to capture an image.

[0153] Aspect 22: A method for processing image data, the method comprising: receiving an image from an image sensor; determining a first image feature score of a first image dataset, the first image dataset corresponding to a first scaling variation of the image at a first scaling intensity; determining a second image feature score of a second image dataset, the second image dataset corresponding to a second scaling variation of the image at a second scaling intensity; identifying that the second image feature score is less than an image feature threshold; and outputting the first image dataset as an output image based on the identification that the second image feature score is less than the image feature threshold.

[0154] Aspect 23: According to the method of aspect 22, wherein outputting the first image dataset is based on the output image also being based on the first image feature score being greater than or equal to the image feature threshold.

[0155] Aspect 24: The method according to any one of Aspects 22 to 23, wherein the first image feature score is a first image sharpness score, the second image feature score is a second image sharpness score, and the image feature threshold is an image sharpness threshold.

[0156] Aspect 25: The method according to any one of aspects 22 to 24, wherein the first image feature score is a first undersharpened value of the first image dataset, the second image feature score is a second undersharpened value of the second image dataset, and the image feature threshold is an undersharpened value threshold.

[0157] Aspect 26: The method according to any one of aspects 22 to 25 further includes: receiving one or more inputs identifying a portion of an image, wherein a first scaling variant and a second scaling variant of the image are based on the identified portion of the image.

[0158] Aspect 27: The method according to aspect 26, wherein the first scaling variant of the image and the second scaling variant of the image include the identified portion of the image.

[0159] Aspect 28: The method according to any one of Aspects 26 or 27, wherein one or more inputs include user input via a user interface.

[0160] Aspect 29: The method according to any one of aspects 26 to 28, wherein one or more inputs include at least one of touch input, gesture input and gaze input.

[0161] Aspect 30: The method according to any one of aspects 26 to 29 further includes: receiving one or more inputs from an object detection algorithm, the object detection algorithm determining that the identified portion of the image includes a depiction of the type of object.

[0162] Aspect 31: According to the method described in aspect 30, the type of the object is a face.

[0163] Aspect 32: The method according to any one of aspects 22 to 31, wherein the one or more processors are configured to: generate a first scaling variant of the image at least in part by cropping and enlarging the image according to a first scaling intensity; and generate a second scaling variant of the image at least in part by cropping and enlarging the image according to a second scaling intensity and at least one of the first scaling variant of the image.

[0164] Aspect 33: The method according to any one of aspects 22 to 32, wherein outputting a first image dataset as an output image includes rendering the output image for display on a display screen.

[0165] Aspect 34: The method according to any one of Aspects 22 to 33, wherein one or more processors are configured to: identify that the difference between a first image feature score and a second image feature score is less than a difference threshold; and output the second image dataset as a second output image based on the difference being less than the difference threshold.

[0166] Aspect 35: The method according to any one of aspects 22 to 34, wherein the first image data set corresponds to a first digitally scaled portion of the image under a first scaling intensity, and the second image data set corresponds to a second digitally scaled portion of the image under a second scaling intensity.

[0167] Aspect 36: According to the method of aspect 35, wherein the second digitally scaled portion of the image is a subset of the first digitally scaled portion of the image.

[0168] Aspect 37: According to the method of aspect 35, wherein the first digital scaling portion of the image is different from the second digital scaling portion of the image.

[0169] Aspect 38: The method according to any one of aspects 22 to 37, wherein outputting a first image dataset as an output image includes sending the output image using a communication interface.

[0170] Aspect 39: A non-transitory computer-readable medium having instructions stored thereon, the instructions causing the one or more processors, when executed, to perform the operations according to any one of 2 to 38.

[0171] Aspect 40: An apparatus for image processing, the apparatus comprising means for performing operations according to any one of aspects 2 to 38.

Claims

1. An apparatus for processing image data, the apparatus comprising: Memory; as well as One or more processors coupled to the memory, the one or more processors being configured to cause the device to: Receive input image data from the image sensor; as well as The input image data is processed to generate an output image for display, wherein, for processing the input image data, the one or more processors are configured to cause the device to: Determine a first image characteristic score that includes a first digital scaling portion of a first image dataset, the first image dataset corresponding to a first digital scaling variant of the image under a first digital scaling intensity; Determine a second image characteristic score that includes a second digital scaling portion of a second image dataset, the second image dataset corresponding to a second digital scaling variant of the image under a second digital scaling intensity; The second image feature score is less than the image feature threshold. as well as The output image is generated using the first digital scaling portion based on the fact that the score of the second image feature is less than the image feature threshold.

2. The apparatus of claim 1, wherein, The output image generated using the first digital scaling portion is also based on the first image feature score being greater than or equal to the image feature threshold.

3. The apparatus of claim 1, wherein, The first image feature score is a first image sharpness score, the second image feature score is a second image sharpness score, and the image feature threshold is an image sharpness threshold.

4. The apparatus of claim 1, wherein, The first image feature score is the first undersharpened value of the first image dataset, the second image feature score is the second undersharpened value of the second image dataset, and the image feature threshold is the undersharpened value threshold.

5. The apparatus of claim 1, wherein, The one or more processors are configured to cause the device to: Receive one or more inputs that identify a portion of the image, wherein the first digital scaling variant of the image and the second digital scaling variant of the image are based on the identified portion of the image.

6. The apparatus of claim 5, wherein, The first digitally scaled portion and the second digitally scaled portion of the image include the identified portion of the image.

7. The apparatus of claim 5, wherein, The one or more inputs include user input via a user interface.

8. The apparatus of claim 7, wherein, The one or more inputs include at least one of touch input, gesture input, and gaze input.

9. The apparatus according to claim 5, wherein, The one or more processors are configured to cause the device to receive the one or more inputs according to an object detection algorithm, the object detection algorithm determining that the identified portion of the image includes a depiction of the type of object.

10. The apparatus according to claim 9, wherein, The type of the object is a face.

11. The apparatus according to claim 1, wherein, The one or more processors are configured to cause the device to: The first digital scaling variant of the image is generated at least in part by cropping and enlarging the image according to the first digital scaling intensity; as well as The second digital scaling variant of the image is generated at least in part by cropping and enlarging the image according to the second digital scaling intensity and at least one of the first digital scaling variant of the image.

12. The apparatus according to claim 1, wherein, The one or more processors are configured to cause the device to render the output image for display on a display screen.

13. The apparatus of claim 12, further comprising: The display screen is configured to display the output image.

14. The apparatus according to claim 1, wherein, The one or more processors are configured to cause the device to: The difference between the first image feature score and the second image feature score is less than a difference threshold; and The second image dataset is output as the second output image based on the difference being less than the difference threshold.

15. The apparatus according to claim 14, wherein, The second digitally scaled portion of the image is a subset of the first digitally scaled portion of the image.

16. The apparatus according to claim 14, wherein, The first digitally scaled portion of the image is different from the second digitally scaled portion of the image.

17. The apparatus according to claim 1, wherein, The one or more processors are configured to enable the device to send the output image using a communication interface.

18. The apparatus of claim 17, further comprising: The communication interface is configured to send the output image to the device.

19. The apparatus according to claim 1, wherein, The device is one of a mobile device, a mobile handheld device, a wireless communication device, a head-mounted display, and a camera.

20. The apparatus according to claim 1, further comprising: The image sensor is configured to capture the image.

21. A method for processing image data, the method comprising: Receive input image data from the image sensor; Processing the input image data to generate an output image for display, wherein processing the input image data includes: Determine a first image characteristic score that includes a first digital scaling portion of a first image dataset, the first image dataset corresponding to a first digital scaling variant of the image under a first digital scaling intensity; Determine a second image characteristic score that includes a second digital scaling portion of a second image dataset, the second image dataset corresponding to a second digital scaling variant of the image under a second digital scaling intensity; The second image feature score is less than the image feature threshold; and The output image is generated using the first digital scaling portion based on the fact that the score of the second image feature is less than the image feature threshold.

22. The method according to claim 21, wherein, The output image generated using the first digital scaling portion is also based on the first image feature score being greater than or equal to the image feature threshold.

23. The method according to claim 21, wherein, The first image feature score is a first image sharpness score, the second image feature score is a second image sharpness score, and the image feature threshold is an image sharpness threshold.

24. The method according to claim 21, wherein, The first image feature score is the first undersharpened value of the first image dataset, the second image feature score is the second undersharpened value of the second image dataset, and the image feature threshold is the undersharpened value threshold.

25. The method of claim 21, further comprising: Receive one or more inputs that identify a portion of the image, wherein the first digital scaling variant of the image and the second digital scaling variant of the image are based on the identified portion of the image.

26. The method of claim 25, wherein, The first digital scaling variant and the second digital scaling variant of the image include the identified portion of the image.

27. The method of claim 21, further comprising: The first digital scaling variant of the image is generated at least in part by cropping and enlarging the image according to the first digital scaling intensity; as well as The second digital scaling variant of the image is generated at least in part by cropping and enlarging the image according to the second digital scaling intensity and at least one of the first digital scaling variant of the image.

28. The method of claim 21, further comprising rendering the output image for display on a display screen.

29. The method of claim 21, further comprising sending the output image using a communication interface.

30. An apparatus for processing image data, the apparatus comprising components for performing the method according to any one of claims 21 to 29.

31. A computer-readable medium having program code recorded thereon, wherein, The program code can be executed by one or more processors to cause the processors to perform the method according to any one of claims 21 to 29.

32. A computer program product comprising computer-readable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 21 to 29.

Images