Content-Based Image Processing

Abstract

To provide a method for sharpening segments of an image differently based on a content in the image.SOLUTION: Sharpening of a pixel of an image based on a content is performed by a content image processing circuit that receives luminance values of the image and a content map. The content map identifies categories of a content in segments of the image. Based on one or more of the identified categories of the content, the circuit determines a content factor associated with the pixel. The content factor may also be based on a texture value and / or a chroma value. The texture value indicates a likelihood of the category of the content and is based on detected edges in the image. The chroma value indicates a likelihood of the category of the content and is based on color information of the image. The circuit receives the content factor and applies it to a version of the luminance value of the pixel to generate a sharpened version of the luminance value.SELECTED DRAWING: None

Description

[Technical field]

[0001] FIELD This disclosure relates to circuitry for processing images, and more particularly to differentially sharpening segments of an image based on content within the image. [Background technology]

[0002] Image data captured by an image sensor or received from other data sources is often processed in an image processing pipeline before further processing or consumption. For example, raw image data may be corrected, filtered, or otherwise modified before being provided to a subsequent component, such as a video encoder. Various components, unit stages, or modules may be used to correct or enhance the captured image data.

[0003] Such image processing pipelines may be configured to advantageously perform corrections or enhancements to captured image data without consuming other system resources. Although many image processing algorithms may be performed by executing software programs on a Central Processing Unit (CPU), executing such programs on the CPU consumes significant CPU bandwidth and other peripheral resources as well as increases power consumption. Thus, image processing pipelines are often implemented as hardware components separate from the CPU that are dedicated to performing one or more image processing algorithms.

[0004] Image processing pipelines often include sharpening or smoothing processes that are performed using one or more adjustment parameters that are applied uniformly across the entire image, so that one or more segments of content within an image may be smoothed or sharpened in a way that adversely affects the appearance of the final image.

[0005] Similarly, other image processing processes such as tone mapping, white balancing, and noise reduction are typically performed using one or more adjustment parameters that are applied uniformly across the entire image, such that one or more segments of content within an image may be processed in a way that adversely affects the appearance of the final image. Summary of the Invention

[0006] Some embodiments relate to an image processor including a content image processing circuit for determining a content coefficient using a content map that identifies a category of content in a segment of an image. The content image processing circuit includes a content coefficient circuit and a content modification circuit coupled to the content coefficient circuit. The content coefficient circuit determines a content coefficient associated with a pixel of the image according to an identified category of content in the image and at least one of a texture value of the pixel in the image or a chroma value of the pixel in the image. The texture value indicates a likelihood of a category of content associated with the pixel based on a texture in the image. The chroma value indicates a likelihood of a category of content associated with the pixel based on color information of the image. The content modification circuit receives the content coefficient from the content coefficient circuit. The content modification circuit generates a sharpened version of the luminance pixel value of the pixel by applying at least the content coefficient to a version of the luminance pixel value of the pixel.

[0007] In some embodiments, the content map is generated by a neural processor circuit that performs machine learning operations on a version of the image to generate the content map.

[0008] In some embodiments, the content map is downscaled relative to the image. The content coefficient circuit may determine the content coefficients by upsampling the content map. The content map may be upsampled by (1) obtaining content coefficients associated with grid points in the content map, and (2) interpolating content coefficients associated with grid points that surround pixels when the content map is scaled up to match the size of the image.

[0009] In some embodiments, the content coefficients are weighted according to a likelihood value, which is based on one of an identified category of content in the image, a texture value, and a chroma value.

[0010] In some embodiments, when a luminance version of an image is split into first information and second information that includes lower frequency components than the frequency components of the first component, the luminance pixel values ​​are included in the first information of the image.

[0011] In some embodiments, the image processor includes a bilateral filter coupled to the content image processing circuitry, the bilateral filter generating a version of the luminance pixel value.

[0012] In some embodiments, the content modification circuitry applies the content coefficient to the version of the luma pixel value by multiplying the content coefficient with the version of the luma pixel value when the content coefficient is above a threshold, and the content modification circuitry applies the content coefficient to the version of the luma pixel value by blending the version of the luma pixel value based on the content coefficient in response to the content coefficient being less than the threshold.

[0013] In some embodiments, the content map is a heat map that indicates the amount of sharpening applied to pixels of the image that correspond to grid points in the content map. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram of an electronic device according to one embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating components of an electronic device according to one embodiment. [Diagram 3] FIG. 2 is a block diagram illustrating an image processing pipeline implemented using an image signal processor according to one embodiment. [Figure 4] FIG. 2 is a block diagram illustrating a portion of an image processing pipeline including a multi-band noise reduction circuit according to one embodiment. [Diagram 5] FIG. 2 is a block diagram illustrating a neural processor circuit that provides a content map to an image signal processor according to one embodiment. [Figure 6A] 1 is an exemplary input image of a neural processor circuit according to one embodiment. [Figure 6B] 6B is a content map based on the input image of FIG. 6A according to one embodiment. [Figure 7] FIG. 2 is a block diagram illustrating components of a first contrast enhancement stage circuit according to one embodiment. [Figure 8] FIG. 2 is a block diagram illustrating components of a content image processing circuit, according to one embodiment. [Figure 9] 1 is a plot illustrating a texture model according to one embodiment. [Figure 10] 1 is a plot illustrating a chroma model according to one embodiment. [Figure 11] FIG. 2 illustrates a method for upsampling a content map according to one embodiment. [Figure 12] 1 is a flow chart illustrating a method for sharpening one or more pixels of an image based on content within a segment of the image, according to one embodiment.

[0015] Various non-limiting embodiments are shown in the drawings and described in the detailed description for purposes of illustration only. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments being described. However, the described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0017] An embodiment of the present disclosure relates to sharpening a segment of an image based on content within the segment as indicated by a content map. A content coefficient for a pixel or segment of the image is determined based on one or more of the identified categories of content associated with the pixel or segment. The content coefficient may also be adjusted based on a set of texture values ​​and / or a set of chroma values. The texture value indicates the likelihood of one of the identified categories of content and is based on the texture within the image. The chroma value indicates the likelihood of one of the identified categories of content and is based on color information of the image. The content coefficient is applied to the pixel or segment to generate a sharpened version of the luminance value.

[0018] Exemplary Electronic Devices Embodiments of electronic devices, user interfaces for such devices, and processes associated with the use of such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a personal digital assistant (PDA) function and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, Apple Watch®, and iPad® devices from Apple Inc. of Cupertino, Calif. Other portable electronic devices, such as wearables, laptops, or tablet computers, are optionally used. In some embodiments, the device is not a portable communication device, but is a desktop computer or other computing device that is not designed for portable use. In some embodiments, the disclosed electronic devices may include a touch-sensitive surface (e.g., a touchscreen display and / or a touchpad). Exemplary electronic devices (e.g., device 100) described below in conjunction with FIG. 1 may include a touch-sensitive surface for receiving user input. The electronic device may also include one or more other physical user interface devices, such as a physical keyboard, a mouse, and / or a joystick.

[0019] 1 is a schematic diagram of an electronic device 100 according to one embodiment. The device 100 may include one or more physical buttons, such as a "home" button or menu button 104. The menu button 104 is used, for example, to navigate to any application in a set of applications running on the device 100. In some embodiments, the menu button 104 includes a fingerprint sensor that identifies a fingerprint on the menu button 104. The fingerprint sensor may be used to determine whether a finger on the menu button 104 has a fingerprint that matches a stored fingerprint to unlock the device 100. Alternatively, in some embodiments, the menu button 104 is implemented as a soft key in a Graphical User Interface (GUI) displayed on a touch screen.

[0020] In some embodiments, device 100 includes a touch screen 150, a menu button 104, a push button 106 for powering the device on / off and locking the device, a volume control button 108, a Subscriber Identity Module (SIM) card slot 110, a headset jack 112, and an external docking / charging port 124. Push button 106 may be used to power the device on / off by pressing and holding the button down for a predefined time interval, lock the device and / or unlock the device or initiate an unlocking process by pressing and releasing the button before the predefined time interval has elapsed. In an alternative embodiment, device 100 also accepts verbal input via microphone 113 to activate or deactivate certain features. Device 100 includes various components, including, but not limited to, memory (which may include one or more computer-readable storage media), a memory controller, one or more central processing units (CPUs), peripheral interfaces, RF circuitry, audio circuitry, speaker 111, microphone 113, an Input / Output (I / O) subsystem, and other input or control devices. Device 100 may include one or more image sensors 164, one or more proximity sensors 166, and one or more accelerometers 168. Device 100 may include two or more types of image sensors 164. Each type may include two or more image sensors 164. For example, one type of image sensor 164 may be a camera and another type of image sensor 164 may be an infrared sensor that may be used for facial recognition. Additionally or alternatively, image sensors 164 may be associated with different lens configurations. For example, device 100 may include a rear image sensor, one with a wide-angle lens and the other with a telephoto lens. Device 100 may include components not shown in FIG. 1, such as an ambient light sensor, a dot projector, and a flood illuminator.

[0021] The device 100 is merely an example of an electronic device, and the device 100 may have more or fewer components than those listed above, and some of the components may be combined into one component or have another configuration or arrangement. The various components of the device 100 listed above may be embodied in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application specific integrated circuits (ASICs). Although the components in FIG. 1 are generally shown as being located on the same side as the touch screen 150, one or more components may be located on the opposite side of the device 100. For example, the front of the device 100 may include an infrared image sensor 164 for face recognition and another image sensor 164 as the front camera of the device 100. The back of the device 100 may also include two additional image sensors 164 as the rear cameras of the device 100.

[0022] FIG. 2 is a block diagram illustrating components of device 100, according to one embodiment. Device 100 may perform various operations, including image processing. For this and other purposes, device 100 may include, among other components, an image sensor 202, a System-ON-a Chip (SOC) component 204, a system memory 230, persistent storage (e.g., flash memory) 228, an orientation sensor 234, and a display 216. The components as shown in FIG. 2 are merely exemplary. For example, device 100 may include other components (such as a speaker or microphone) not shown in FIG. 2. Additionally, some components (such as orientation sensor 234) may be omitted from device 100.

[0023] The image sensors 202 are components for capturing image data. Each of the image sensors 202 is a component for capturing image data and may be embodied as, for example, a Complementary Metal-Oxide-Semiconductor (CMOS) active pixel sensor, a camera, a video camera, or other devices. The image sensors 202 generate raw image data that is sent to the SOC components 204 for further processing. In some embodiments, the image data processed by the SOC components 204 is displayed on the display 216, stored in the system memory 230, the persistent storage 228, or sent to a remote computing device via a network connection. The raw image data generated by the image sensors 202 may be a Bayer Color Filter Array (CFA) pattern (hereinafter also referred to as a "Bayer pattern"). The image sensors 202 may also include optical and mechanical components that assist the image sensing components (e.g., pixels) in capturing an image. The optical and mechanical components may include an aperture, a lens system, and an actuator that controls the focal length of the image sensor 202 .

[0024] The motion sensor 234 is a component or set of components for sensing the movement of the device 100. The motion sensor 234 may generate a sensor signal indicative of the orientation and / or acceleration of the device 100. The sensor signal is sent to the SOC component 204 for various operations, such as turning on the device 100 or rotating an image displayed on the display 216.

[0025] The display 216 is a component for displaying images generated by the SOC components 204. The display 216 may include, for example, a Liquid Crystal Display (LCD) device or an Organic Light Emitting Diode (OLED) device. Based on the data received from the SOC components 204, the display 116 may display various images, such as menus, selected operating parameters, images captured by the image sensor 202 and processed by the SOC components 204, and / or other information received from a user interface (not shown) of the device 100.

[0026] The system memory 230 is a component for storing instructions for execution by the SOC components 204 and data processed by the SOC components 204. The system memory 230 may be embodied as any type of memory including, for example, Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM), Static RAM (SRAM), or a combination thereof. In some embodiments, the system memory 230 may store pixel data or other image data, or statistical data in various formats.

[0027] The persistent storage 228 is a component for storing data in a non-volatile manner. The persistent storage 228 retains data even in the absence of power. The persistent storage 228 may be embodied as a Read-Only Memory (ROM), a flash memory, or other non-volatile random access memory device. The persistent storage 228 stores the operating system and various software applications of the device 100. The persistent storage 228 may also store one or more machine learning models, such as regression models, Support Vector Machines (SVMs), such as a random forest model, a kernel SVM, and Artificial Neural Networks (ANNs), such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, and Long Short Term Memory (LSTMs). The machine learning models may be independent models that operate in conjunction with the neural processor circuit 218 and various software applications or sensors of the device 100. The machine learning models may be part of the software applications. Machine learning models may perform a variety of tasks such as facial recognition, image classification, object classification, concept classification, and information classification, speech recognition, machine translation, voice recognition, voice command recognition, text recognition, text and context analysis, other natural language processing, predictions and recommendations.

[0028] The various machine learning models stored on device 100 may be fully trained, untrained, or partially trained to allow device 100 to continue to enhance or train the machine learning models as device 100 is used. The operation of a machine learning model includes various calculations used in training the model and using the model to determine results at run time. For example, in one case, device 100 captures a facial image of a user and uses the image to continue to improve the machine learning model used to lock or unlock device 100.

[0029] The SOC component 204 is embodied as one or more Integrated Circuit (IC) chips and performs various data processing processes. The SOC component 204 may include, among other subcomponents, an Image Signal Processor (ISP) 206, a Central Processing Unit (CPU) 208, a network interface 210, a motion sensor interface 212, a display controller 214, a Graphics Processor (GPU) 220, a memory controller 222, a video encoder 224, a storage controller 226, a neural processor circuit 218, and a bus 232 connecting these subcomponents. The SOC component 204 may include more or fewer subcomponents than those shown in FIG. 2.

[0030] The ISP 206 is a circuit that performs various stages of an image processing pipeline. In some embodiments, the ISP 206 may receive raw image data from the image sensor 202 and process the raw image data into a form that can be used by other subcomponents of the SOC component 204 or components of the device 100. The ISP 206 may perform various image manipulation operations, such as image transformation operations, horizontal and vertical scaling, color space conversion, and / or image stabilization conversion, as described in more detail below with reference to FIG.

[0031] CPU 208 may be embodied using any suitable instruction set architecture and may be configured to execute instructions defined in that instruction set architecture. CPU 208 may be a general-purpose or embedded processor using any of a variety of ISAs, such as x86, PowerPC, SPARC, RISC, ARM, or MIPS Instruction Set Architectures (ISAs), or any other suitable ISA. Although a single CPU is shown in FIG. 2, SOC component 204 may include multiple CPUs. In a multiprocessor system, each of the CPUs may commonly implement the same ISA, although this is not necessarily the case.

[0032] Graphics Processing Unit (GPU) 220 is a graphics processing circuit for performing operations on graphics data. For example, GPU 220 may render objects (e.g., those containing pixel data for an entire frame) to be displayed in a frame buffer. GPU 220 may include one or more graphics processors that may run graphics software or hardware acceleration of certain graphics operations to perform some or all of the graphics operations.

[0033] The neural processor circuit 218 is a programmable circuit that performs machine learning operations on the input data of the neural processor circuit 218. The machine learning operations may include different calculations for training a machine learning model and for performing inference or prediction based on the trained machine learning model. The neural processor circuit 218 is a circuit that performs various machine learning operations based on calculations including multiplication, addition, and accumulation. Such calculations may be organized to perform various types of tensor multiplications, such as tensor products and convolutions of input data and kernel data. The neural processor circuit 218 is a configurable circuit that performs these operations quickly and power-efficiently while relieving the CPU 208 from resource-intensive operations associated with neural network operations. The neural processor circuit 218 may receive input data from the sensor interface 212, the image signal processor 206, the persistent storage 228, the system memory 230, or other sources such as the network interface 210 or the GPU 220. The output of the neural processor circuitry 218 may be provided to various components of the device 100, such as the image signal processor 206, the system memory 230, or the CPU 208 for various operations.

[0034] The network interface 210 is a sub-component that allows data to be exchanged between the device 100 and other devices (e.g., carrier devices or agent devices) over one or more networks. For example, video or other image data may be received from other devices over the network interface 210 and stored in the system memory 230 for subsequent processing and display (e.g., via a back-end interface to the image signal processor 206 as described below with respect to FIG. 3). The networks may include, but are not limited to, a Local Area Network (LAN) (e.g., an Ethernet or enterprise network) and a Wide Area Network (WAN). The image data received over the network interface 210 may be subjected to an image processing process by the ISP 206.

[0035] The motion sensor interface 212 is circuitry for interfacing with the motion sensor 234. The motion sensor interface 212 receives sensor information from the motion sensor 234 and processes the sensor information to determine the orientation or motion of the device 100.

[0036] The display controller 214 is a circuit for transmitting image data to be displayed on the display 216. The display controller 214 receives image data from the ISP 206, the CPU 208, the graphics processor, or the system memory 230, and processes the image data into a format suitable for display on the display 216.

[0037] The memory controller 222 is a circuit for communicating with the system memory 230. The memory controller 222 can cause the ISP 206, the CPU 208, the GPU 220, or other subcomponents of the SOC component 204 to read data from the system memory 230 for processing. The memory controller 222 can also write data received from various subcomponents of the SOC component 204 to the system memory 230.

[0038] The video encoder 224 is hardware, software, firmware, or a combination thereof that encodes the video data into a format suitable for storage in persistent storage 128 or passes the data to the network interface 210 for transmission over a network to another device.

[0039] In some embodiments, one or more subcomponents of the SOC component 204, or the functionality of some of these subcomponents, may be performed by software components executing on the neural processor circuitry 218, the ISP 206, the CPU 208, or the GPU 220. Such software components may be stored in the system memory 230, the persistent storage 228, or another device in communication with the device 100 via the network interface 210.

[0040] Image data or video data may flow through various data paths within the SOC component 204. In one embodiment, raw image data may be generated from the image sensor 202, processed by the ISP 206, and then sent to the system memory 230 via the bus 232 and the memory controller 222. After the image data is stored in the system memory 230, it may be accessed via the bus 232 by the video encoder 224 for encoding or by the display 116 for display.

[0041] In another embodiment, image data is received from a source other than image sensor 202. For example, video data may be streamed, downloaded, or otherwise communicated to SOC component 204 via a wired or wireless network. Image data may be received via network interface 210 and written to system memory 230 via memory controller 222. The image data may then be retrieved from system memory 230 by ISP 206 and processed through one or more image processing pipeline stages, as described in more detail below with reference to FIG. 3. The image data may then be returned to system memory 230 or sent to video encoder 224, display controller 214 (for display on display 216), or storage controller 226 for storage in persistent storage 228.

[0042] Exemplary Image Signal Processing Pipeline FIG. 3 is a block diagram illustrating an image processing pipeline implemented with ISP 206, according to one embodiment. In the embodiment of FIG. 3, ISP 206 is coupled to image sensor system 201, which includes one or more image sensors 202A-202N (hereinafter collectively referred to as “image sensors 202” or individually referred to as “image sensors 202”), to receive raw image data. Image sensor system 201 may include one or more subsystems that individually control image sensors 202. In some cases, each image sensor 202 can operate independently, and in other cases, image sensors 202 can share some components. For example, in one embodiment, two or more image sensors 202 may share the same circuit board that controls the mechanical components of the image sensors (e.g., actuators that change the focal length of each image sensor). The image sensing components of image sensor 202 may include different types of image sensing components that may provide raw image data in different forms to ISP 206. For example, in one embodiment, the image sensing components may include a number of focus pixels used for autofocus and a number of image pixels used to capture an image. In another embodiment, an image sensing pixel may be used for both autofocus purposes and image capture purposes.

[0043] ISP 206 implements an image processing pipeline that may include a set of stages that process image information from creation, capture, or receipt to output. ISP 206 may include, among other components, a sensor interface 302, a central control module 320, a front-end pipeline stage 330, a back-end pipeline stage 340, an image statistics module 304, a scaler 322, a back-end interface 342, an output interface 316, and auto-focus circuits 350A-350N (hereinafter collectively referred to as "auto-focus circuits 350" or individually referred to as "auto-focus circuits 350"). ISP 206 may include other components not shown in FIG. 3 or may omit one or more components shown in FIG. 3.

[0044] In one or more embodiments, different components of the ISP 206 process image data at different rates. In the embodiment of FIG. 3, the front-end pipeline stages 330 (e.g., the RAW processing stage 306 and the resampling processing stage 308) may process image data at an initial rate. Thus, various different techniques, adjustments, corrections, or other processing operations are performed by these front-end pipeline stages 330 at the initial rate. For example, if the front-end pipeline stages 330 process two pixels per clock cycle, the operations of the RAW processing stage 306 (e.g., black level compensation, highlight recovery, and bad pixel correction) may process two pixels of image data simultaneously. In contrast, one or more back-end pipeline stages 340 may process image data at different speeds that are less than the initial data rate. For example, in the embodiment of FIG. 3, the back-end pipeline stages 340 (e.g., the noise processing stage 310, the color processing stage 312, and the output rescaling module 314) may be processed at a reduced rate (e.g., one pixel per clock cycle).

[0045] The raw image data captured by the image sensor 202 may be transmitted in different ways to different components of the ISP 206. In one embodiment, the raw image data corresponding to focus pixels may be transmitted to the autofocus circuitry 350, while the raw image data corresponding to image pixels may be transmitted to the sensor interface 302. In another embodiment, the raw image data corresponding to both types of pixels may be transmitted to both the autofocus circuitry 350 and the sensor interface 302 simultaneously.

[0046] The autofocus circuitry 350 may include hardware circuitry that analyzes the raw image data to determine the appropriate focal length for each image sensor 202. In one embodiment, the raw image data may include data transmitted from image sensing pixels dedicated to image focusing. In another embodiment, the raw image data from the image capture pixels may also be used for autofocus purposes. The autofocus circuitry 350 may perform various image processing operations to generate data that determines the appropriate focal length. The image processing operations may include cropping, binning, image compensation, scaling to generate data used for autofocus purposes. The autofocus data generated by the autofocus circuitry 350 may be fed back to the image sensor system 201 to control the focal length of the image sensor 202. For example, the image sensor 202 may include control circuitry that analyzes the autofocus data to determine a command signal to be transmitted to an actuator associated with the image sensor's lens system to change the focal length of the image sensor. The data generated by the autofocus circuitry 350 may also be transmitted to other components of the ISP 206 for other image processing purposes. For example, a portion of the data may be transmitted to the image statistics module 304 to determine information regarding autoexposure.

[0047] The autofocus circuit 350 may be an individual circuit separate from other components such as the image statistics module 304, the sensor interface 302, the front-end module 330, and the back-end module 340. This allows the ISP 206 to perform autofocus analysis independently from other image processing pipelines. For example, the ISP 206 may analyze raw image data from the image sensor 202A to adjust the focal length of the image sensor 202A using the autofocus circuit 350A while performing downstream image processing of the image data from the image sensor 202B. In one embodiment, the number of autofocus circuits 350 may correspond to the number of image sensors 202. In other words, each image sensor 202 may have a corresponding autofocus circuit dedicated to autofocusing the image sensor 202. The device 100 may perform autofocus on different image sensors 202 even if one or more image sensors 202 are not actively used. This allows for a seamless transition between the two image sensors 102 when the device 100 switches from one image sensor 202 to another. For example, in one embodiment, the device 100 may include a wide-angle camera and a telephoto camera as a dual back camera system for photographing and processing images. The device 100 may display images captured by one of the dual cameras and may switch between the two cameras from time to time. Since two or more autofocus circuits 350 may continuously provide autofocus data to the image sensor system 201, the displayed image may seamlessly transition from image data captured by one image sensor 202 to image data captured by another image sensor without waiting for the second image sensor 202 to adjust its focal length.

[0048] Raw image data captured by different image sensors 202 may also be sent to the sensor interface 302. The sensor interface 302 receives raw image data from the image sensor 202 and processes the raw image data into image data that can be processed by other stages in the pipeline. The sensor interface 302 may perform various pre-processing operations such as image cropping, binning, or scaling to reduce image data size. In some embodiments, pixels are sent from the image sensor 202 to the sensor interface 302 in raster order (e.g., horizontally, row by row). Subsequent processes in the pipeline may also be performed in raster order, and the results may also be output in raster order. Although only one image sensor and one sensor interface 302 are shown in FIG. 3, if more than one image sensor is provided in the device 100, a corresponding number of sensor interfaces may be provided in the ISP 206 to process the raw image data from each image sensor.

[0049] The front-end pipeline stage 330 processes image data in the RAW color domain or the full color domain. The front-end pipeline stage 330 may include, but is not limited to, a RAW processing stage 306 and a resampling processing stage 308. The RAW image data may be, for example, in a Bayer RAW format. In the Bayer RAW image format, each pixel is provided with pixel data having a specific value for a specific color (rather than for all colors). In an image capture sensor, the image data is typically provided in a Bayer pattern. The RAW processing stage 306 may process the image data in the Bayer RAW format.

[0050] The operations performed by the RAW processing stage 306 include, but are not limited to, sensor linearization, black level compensation, fixed pattern noise reduction, bad pixel correction, RAW noise filtering, lens shading compensation, white balance gain, and highlight recovery. Sensor linearization refers to mapping nonlinear image data into a linear space for further processing. Black level compensation refers to providing digital gain, offset, and clip independently for each color component (e.g., Gr, R, B, Gb) of the image data. Fixed pattern noise reduction refers to removing offset and gain fixed pattern noise by subtracting a dark frame from the input image and multiplying the pixels with different gains. Bad pixel correction refers to detecting bad pixels and then replacing the bad pixel value. RAW noise filtering refers to reducing noise in the image data by averaging adjacent pixels of similar luminance. Highlight recovery refers to estimating pixel values ​​for pixels that are clipped (or nearly clipped) from other channels. Lens shading compensation refers to applying per-pixel gain to compensate for the reduction in luminosity approximately proportional to the distance from the optical center of the lens. White balance gain refers to providing digital gain, offset and clip for white balance independently for all color components (e.g., Gr, R, B, Gb in Bayer format). The components of the ISP 206 can convert RAW image data to full color domain image data, and thus the RAW processing stage 306 can process full color domain image data in addition to or instead of the RAW image data.

[0051] The resampling stage 308 performs various operations to convert, resample, or scale the image data received from the RAW processing stage 306. The operations performed by the resampling stage 308 may include, but are not limited to, a demosaicing operation, a per-pixel color correction operation, a gamma mapping operation, color space conversion, and downscaling or subband splitting. The demosaicing operation refers to converting or interpolating missing color samples from the RAW image data (e.g., to a Bayer pattern) to output the image data to the full color domain. The demosaicing operation may include directional low-pass filtering on the interpolated samples to obtain full color pixels. The per-pixel color correction operation refers to the process of performing color correction on a pixel-by-pixel basis using information about the relative noise standard deviation of each color channel to correct the color without amplifying the noise in the image data. The gamma mapping refers to converting the image data from input image data values ​​to output data values ​​and performing gamma correction. For the purpose of gamma mapping, a lookup table (or other structure that indexes pixel values ​​to another value) for the different color components or color channels of each pixel (e.g., separate lookup tables for R, G, and B color components) can be used. Color space conversion refers to converting the color space of the input image data to a different format. In one embodiment, the resampling processing stage 308 converts the RGG format to a YCbCr format for further processing. In one embodiment, the resampling processing stage 308 converts the RBD format to a RGB format for further processing.

[0052] The central control module 320 may control and coordinate the overall operation of the other components in the ISP 206. The central control module 320 performs operations including, but not limited to, monitoring various operating parameters (e.g., logging clock cycles, memory latency, quality of service, and status information) to control the starting and stopping of the other components of the ISP 206, updating or managing the control parameters of the other components of the ISP 206, and interfacing with the sensor interface 302. For example, the central control module 320 may update the programmable parameters of the other components in the ISP 206 while the other components are in an idle state. After updating the programmable parameters, the central control module 320 may place these components of the ISP 206 in a running state to perform one or more operations or tasks. The central control module 320 may also instruct the other components of the ISP 206 to store image data (e.g., by writing to the system memory 230 of FIG. 2) before, during, or after the resampling processing stage 308. In this manner, the image data output from the resampling processing stage 308 may be stored as full resolution image data in RAW color domain format or full color domain format in addition to or instead of being processed through the backend pipeline stage 340.

[0053] The image statistics module 304 performs various operations to collect statistical information associated with the image data. The operations for collecting statistical information may include, but are not limited to, sensor linearization, replacement of patterned bad pixels, subsampling of raw image data, detection and replacement of non-patterned bad pixels, black level compensation, lens shading correction, and inverse black level compensation. After performing one or more of such operations, statistical information such as 3A statistics (Auto White Balance (AWB), Auto Exposure (AE), histograms (e.g., 2D color or composition), and any other image data information may be collected or tracked. In some embodiments, a particular pixel value or area of ​​pixel values ​​may be excluded from collecting a particular statistical data if a preceding operation identifies a clipped pixel. Although only one statistics module 304 is shown in FIG. 3, multiple image statistics modules may be included in the ISP 206. For example, each image sensor 202 may correspond to an individual image statistics unit 304. In such an embodiment, each statistics module may be programmed by the central control module 320 to collect different information for the same or different image data.

[0054] The scaler 322 receives the image data and generates a downscaled version of the image. Thus, the scaler 322 can provide a reduced resolution image to various components, such as the neural processor circuit 218. Although the scaler 322 is coupled to the RAW processing stage 306 in FIG. 3, the scaler 322 may also be coupled to receive input images from other components of the image signal processor 206.

[0055] The back-end interface 342 receives image data from image sources other than the image sensor 102 and forwards it to other components of the ISP 206 for processing. For example, the image data may be received over a network connection and stored in the system memory 230. The back-end interface 342 retrieves image data stored in the system memory 230 and provides it to the back-end pipeline stages 340 for processing. One of the many operations performed by the back-end interface 342 is to convert the retrieved image data into a format that can be utilized by the back-end processing stages 340. For example, the back-end interface 342 may convert image data formatted in RGB, YCbCr 4:2:0, or YCbCr 4:2:2 to YCbCr 4:4:4 color format.

[0056] The back-end pipeline stages 340 process image data according to a particular full-color format (e.g., YCbCr 4:4:4 or RGB). In some embodiments, components of the back-end pipeline stages 340 can convert the image data to a particular full-color format before further processing. The back-end pipeline stages 340 can include, among other stages, a noise processing stage 310 and a color processing stage 312. The back-end pipeline stages 340 can include other stages not shown in FIG. 3.

[0057] The noise processing stage 310 performs various operations to reduce noise in the image data. Operations performed by the noise processing stage 310 include, but are not limited to, color space conversion, gamma / degamma mapping, temporal filtering, noise filtering, luma sharpening, and chroma noise reduction. Color space conversion may convert image data from one color space format to another (e.g., converting RGB format to YCbCr format). Gamma / degamma operations convert image data from input image data values ​​to output data values ​​to perform gamma correction or inverse gamma correction. Temporal filtering filters noise using a previously filtered image frame to reduce noise, for example combining pixel values ​​of a previous image frame with pixel values ​​of a current image frame. Noise filtering may include, for example, spatial noise filtering. Luma sharpening may sharpen the luma values ​​of pixel data, while chroma suppression may attenuate chroma to gray (e.g., no color). In some embodiments, luma sharpening and chroma suppression may be performed simultaneously with spatial noise filtering. The aggressiveness of noise filtering may be determined differently for different regions of an image. Spatial noise filtering may be included as part of the time loop that performs the temporal filtering. For example, a previous image frame may be processed by a temporal filter and a spatial noise filter before being stored as a reference frame for the next image frame to be processed. In other embodiments, spatial noise filtering may not be included as part of the time loop for temporal filtering (e.g., a spatial noise filter may be applied to an image frame after it is stored as a reference image frame, and thus the reference frame is not spatially filtered).

[0058] The color processing stage 312 may perform various operations associated with adjusting color information in the image data. Operations performed in the color processing stage 312 include, but are not limited to, local tone mapping, gain / offset / clip, color correction, 3D color lookup, gamma conversion, and color space conversion. Local tone mapping refers to spatially varying a local tone curve to provide more control when rendering an image. For example, a two-dimensional grid of tone curves (which may be programmed by the central control module 320) may be bilinearly interpolated such that a flatly varying tone curve is generated across the image. In some embodiments, local tone mapping may also apply a spatially-varying and luminosity-varying color correction matrix that may be used, for example, to make the sky bluer while darkening the blue of the shadows in the image. Digital gain / offset / clip may be provided for each color channel or color component of the image data. Color correction may apply a color correction transformation matrix to the image data. 3D color lookup may utilize a three-dimensional array of color component output values ​​(e.g., R, G, B) to perform advanced tone mapping, color space conversion, and other color transformations. Gamma conversion may be performed by mapping input image data values ​​to output data values, for example, to perform gamma correction, tone mapping, or histogram matching. Color space conversion may be performed to convert image data from one color space to another (e.g., from RGB to YCbCr). Other processing techniques may also be performed as part of the color processing stage 312 to perform other special image effects, including black and white conversion, sepia tone conversion, negative conversion, or solarization conversion.

[0059] The output rescaling module 314 can resample, transform, and correct distortions on the fly as the ISP 206 processes the image data. The output rescaling module 314 can calculate fractional input coordinates for each pixel and use the fractional coordinates to interpolate the output pixels via a polyphase resampling filter. The fractional input coordinates can be generated from various possible transformations of the output coordinates, such as image resizing or cropping (e.g., via simple horizontal and vertical scaling transformations), image rotation and shearing (e.g., via a non-separable matrix transformation), perspective warping (e.g., via an additional depth transformation) and pixel-by-pixel perspective division applied piecewise to strips due to changes in the image sensor during capture of the image data (e.g., due to a rolling shutter), and geometric distortion correction (e.g., via calculating the radial distance from the optical center to index an interpolated radial gain table and applying a radial perturbation to the coordinates due to the radial distortion of the lens).

[0060] The output rescaling module 314 can apply transformations to the image data as it is processed in the output rescaling module 314. The output rescaling module 314 can include horizontal and vertical scaling components. The vertical portion of the design can implement a series of image data line buffers to hold the "support" required by the vertical filter. Because the ISP 206 can be a streaming device, only lines of image data within a sliding window of lines of finite length may be available to the filter. If a line is discarded to make room for an incoming line, it may become unavailable. The output rescaling module 314 can statistically monitor the calculated input Y coordinates of previous lines and use it to calculate the optimal set of lines to keep within the vertical support window. For each subsequent line, the output rescaling module can automatically generate a guess as to the center of the vertical support window. In some embodiments, the output rescaling module 314 may implement a table of piecewise perspective transformations encoded as a Digital Difference Analyzer (DDA) stepper that performs a pixel-by-pixel perspective transformation between the input image data and the output image data to correct for artifacts and motion caused by sensor movement during capture of the image frames. The output rescaling may provide the image data to various other components of the device 100 via the output interface 316, as described above with respect to Figures 1 and 2.

[0061] In various embodiments, the functionality of components 302-350 may be performed in an order different from that implied by the order of these functional units in the image processing pipeline shown in Figure 3, or may be performed by different functional components than those shown in Figure 3. Additionally, the various components as described in Figure 3 may be embodied in various combinations of hardware, firmware, or software.

[0062] Exemplary Pipeline Associated with a Multi-Band Noise Reduction Circuit Figure 4 is a block diagram illustrating a portion of an image processing pipeline including a Multiple Band Noise Reduction (MBNR) circuit 420, according to one embodiment. In the embodiment of Figure 4, the MBNR circuit 420 is part of a resampling processing stage 308, which also includes, among other components, a scaler 410 and a sub-band splitter circuit 430. The resampling processing stage 308 performs scaling, noise reduction, and sub-band splitting recursively.

[0063] As a result of the recursive process, the resampling stage 308 outputs a series of high frequency component image data HF(N) and low frequency component image data LF(N) derived from the original input image 402, where N represents the level of downsampling performed on the original input image 402. For example, HF(0) and LF(0) represent the high frequency component image data and low frequency component image data, respectively, separated from the original input image 402, and HF(1) and LF(1) represent the high frequency component image data and low frequency component image data, respectively, separated from the first downscaled version of the input image 402.

[0064] The MBNR circuit 420 is a circuit that performs noise reduction on multiple bands of the input image 402. The input image 402 is first passed to the MBNR circuit 420 via a multiplexer 414 for noise reduction. A noise-reduced version 422 of the original input image 402 is generated by the MBNR circuit 420 and provided to a subband splitter 430. The subband splitter 430 splits the noise-reduced version 422 of the original input image 402 into high frequency component image data HF(0) and low frequency component image data LF(0). The high frequency component image data HF(0) is passed to a subband processing pipeline 448 and then to the subband merger 352. In contrast, the low frequency component image LF(0) passes through a demultiplexer 440 and is fed back to the resampling processing stage 308 for downscaling by the scaler 410.

[0065] The scaler 410 generates a downscaled version 412 of the low frequency component image LF(0) fed to the scaler 410, which is passed to the MBNR circuit 420 via a multiplexer 414 for noise reduction. The MBNR circuit 420 performs noise reduction to generate a noise reduced version 432 of the downscaled image 412, which is sent to the subband splitter 430, which splits the processed low frequency image data LF(0) again into high frequency component image data HF(1) and low frequency component image data LF(1). The high frequency component image data HF(1) is sent to the subband processing pipeline 448 and then to the subband merger 352, while the low frequency component image data LF(1) is again fed back to the scaler 410 to repeat the process within the resampling processing stage 308. The process of generating high frequency component image data HF(N) and low frequency component image data LF(N) is repeated until the final level of band splitting by the subband splitter 430 is performed. Upon reaching the final level of band decomposition, the low frequency component image data LF(N) is passed via a demultiplexer 440 and a multiplexer 446 to a subband processing pipeline 448 and a subband merger 352 .

[0066] As described above, the MBNR circuit 420 performs noise reduction on the input image 402 and a downscaled low frequency version of the input image 402. This allows the MBNR circuit 420 to perform noise reduction on multiple bands of the original input image 402. However, it should be noted that without subband splitting and scaling, only a single pass of noise reduction on the input image 402 can be performed by the MBNR circuit 420.

[0067] The subband merger 352 merges the processed high frequency component image data HF(N)' with the processed low frequency component image data LF(N)' to generate a processed LF(N-1)'. The processed LF(N-1)' is fed back to the subband merger 352 via the demultiplexer 450 and the multiplexer 446 for merging with the processed HF(N-1)' to generate a processed LF(N-2)'. The process of merging the processed high frequency component image data with the processed low frequency component data is repeated until the subband merger 352 generates a processed version of the input image 454 which is output via the demultiplexer 450.

[0068] The first contrast enhancement stage 450 and the second contrast enhancement stage 452 perform sharpening or smoothing operations on segments of image data based on content associated with the segments of the image. The first contrast enhancement stage 450 is a component of the subband processing pipeline 448 and performs a sharpening operation on the downscaled high frequency component image data HF(N) for the input image 402, while the second contrast enhancement stage 452 performs sharpening on the output of the subband merger 352, which may be full resolution image data having the same spatial size as the input image 402. The first and second contrast enhancement stages 450, 452 are further described with reference to FIG. 7.

[0069] Example Pipeline Related to Content Map 5 is a block diagram illustrating the provision of a content map 504 (also referred to herein as a "segmentation map") by the neural processor circuit 218 to the image signal processor 206, according to one embodiment. The image signal processor 206 provides an input image 502 to the neural processor circuit 218. The input image 502 may be the same or different compared to the original input image 402. Based on the input image 502, the neural processor circuit 218 generates a content map 504 and provides the content map 504 to the image signal processor 206. For example, the content map 504 is sent to a contrast enhancement stage 450, 452.

[0070] As previously described with reference to FIG. 2, the neural processor circuit 218 may be machine-learned. Thus, the neural processor circuit 218 may determine the content map 504 by performing one or more machine-learning operations on the input image 502. In some embodiments, the input image 502 is a reduced-resolution image (e.g., provided by the scaler 322) compared to the original input image 402. Reducing the resolution of the image may reduce the processing time for generating the content map 504. However, since the resolution of the content map 504 is typically the same or similar to the resolution of the input image 502, the content map 504 may be downscaled relative to the original input image 402. An exemplary input image 602 is provided in FIG. 6A, and an exemplary content map 604 is provided in FIG. 6B. The content map 604 identifies grass 506 and people 508 in the image 602.

[0071] Each segment of content may be associated with one or more predefined categories of content. The content map 504 may be associated with a grid having multiple grid points used during the upscaling process. The content map 504 may be the same size as the full-scale image. In some embodiments, to facilitate various processing, the number of grid points is less than the number of pixels in the full-scale image to be sharpened. In one or more embodiments, each grid point of the map may be associated with a category of content (also referred to herein as a "content category") and may be used to determine content coefficients of nearby pixels in the full-scale image, as described in more detail below with reference to FIG. 11.

[0072] The example content categories in the content map 504 have different desired content coefficients. The different content categories may include skin, leaves, grass, and sky. It is generally desirable to sharpen certain categories (e.g., leaves and grass) relative to other categories (e.g., skin). In one or more embodiments, the neural processor circuit 218 is trained using various machine learning algorithms to classify different segments of the input image 502 and identify the content categories of the content in the input image 502, which is then used by the image signal processor 206 to sharpen different segments of the input image 502 with different content coefficients that indicate the degree of sharpening to be applied to the different segments, as described in more detail below with reference to FIG. 8.

[0073] In other embodiments, the neural processor circuit 218 is trained using various machine learning algorithms to generate a heat map as the content map 504. The heat map does not indicate the content categories associated with different segments, but rather directly indicates the degree of sharpening desired in different segments of the input image 502.

[0074] Example of a contrast enhancement stage circuit FIG. 7 is a block diagram illustrating components of a first contrast enhancement stage circuit 450 according to one embodiment. The first contrast enhancement stage circuit 450 performs content-based image sharpening and smoothing on the luminance information Y to generate a sharpened version of the luminance information Y′. The luminance information Y refers to an image containing only the luminance component of the input image 402, and the sharpened luminance information Y′ refers to an image containing only the luminance component of the output image. The first contrast enhancement stage circuit 450 may include an image sharpener 702, a content image processing 704, and a summing circuit 706, among other components. The second contrast enhancement stage circuit 452 has substantially the same structure as the first contrast enhancement stage circuit 450, except that the luminance image is not downscaled relative to the full input image 402, and therefore, for the sake of brevity, a detailed description thereof is omitted herein.

[0075] The image sharpener 702 is a circuit that performs contrast enhancement (e.g., sharpening) on ​​the luminance information Y and generates an output delta Y. Delta Y represents a mask of Y. For example, delta Y is the result of an unsharp masking process. In one or more embodiments, the image sharpener 702 is embodied as a bilateral filter or a high-pass frequency filter that performs operations on the luminance information Y. Thus, for example, delta Y may be the high frequency components of the image. Delta Y may be further adjusted by components downstream of the first contrast enhancement stage 450.

[0076] Content image processing 704 is a circuit that adjusts delta Y based on the content category identified by the content map and the likelihood of such classification. Content image processing 704 receives the luminance information Y and the content map 504 and generates an adjusted delta Y' that is increased or decreased relative to delta Y depending on the amount of sharpening desired based on the content category, as will be further described with respect to FIG.

[0077] In some embodiments, the summing circuit 706 adds the adjusted delta Y' from the content image processing 704 to the luma information Y to generate the sharpened luma information Y'. In some embodiments, the summing circuit 706 adds the adjusted delta Y' to the low frequency components of the luma information Y (e.g., where low frequency components = Y - delta Y). For some pixels, the adjusted delta Y' is positive, whereby the summing in the summing circuit 706 results in sharpening of the associated segment of the image. For pixels with an adjusted delta Y' that is negative, the summing circuit 706 performs a blurring operation, such as an alpha blur. As explained further below, the delta Y may be alpha blurred with the low frequency components of the luma information Y. This may result in the blurring being limited to the low frequency components. This may prevent image artifacts that may occur when the delta Y' includes large negative values.

[0078] <Example of content image processing circuit> Figure 8 is a block diagram illustrating components of content image processing 704, according to one embodiment. As discussed above with reference to Figure 7, content image processing 704 performs sharpening and smoothing operations based on the content in the image. Content image processing 704 may include a texture circuit 802, a chroma circuit 804, a content coefficient circuit 806, and a content modification circuit 810, among other components.

[0079] The content coefficient circuit 806 determines a content coefficient for a pixel in the input image. A content coefficient may be determined for each pixel in the input image. The content coefficient is based on one or more values ​​in the content map and indicates an amount of sharpening to be applied to the pixel. The content coefficient may also be based on one or more texture values ​​from the texture circuit 802 and / or chroma values ​​from the chroma circuit 804.

[0080] As mentioned above, the content categories of the content map may be associated with content coefficients. For example, a content coefficient is predefined for each content category. If the content map has the same resolution as the image, the content coefficient for a pixel may be obtained by referencing information of the corresponding position in the content map. If the content map is downscaled compared to the image, the content map may be upscaled to match the size of the input image so that the content coefficient may be interpolated from nearby pixels in the upscaled version of the content map. A grid having a number of grid points may be overlaid on the input image, and the information associated with the grid points may be used to determine information of a pixel in the full image by interpolation. For example, if a pixel location in the full image does not match a grid point in the content map (e.g., the pixel is located between a set of grid points), the content coefficient of the pixel may be determined by interpolating the content coefficients of the grid points closest to (e.g., surrounding) the pixel. Upsampling of the content map to determine the content coefficient of pixels between the grid points is further described with respect to FIG. 11.

[0081] In some embodiments, the content coefficients are weighted according to the likelihood value, for example, the content coefficient Q of a pixel is determined by multiplying an initial content coefficient Q0 by the likelihood value. Q=(Q0) * (Likelihood value) (1) Here, the initial content coefficient Q0 is a content coefficient associated with a particular content category. The likelihood value may be based on the content category of the pixel, the texture value and / or the chroma value of the pixel, as described below with reference to the texture circuit 802 and the chroma circuit 804. In some embodiments, the likelihood value is determined by a likelihood model such as: (Likelihood value)=C1+C2 * (Texture value)+C3 * (Chroma value) (2) where C1, C2, and C3 are predefined constants (e.g., tuning parameters). The predefined constants may have values ​​based on the content categories in the content map. In some embodiments, the likelihood model is a polynomial function on the texture and chroma values. The likelihood model represents a model for the accuracy of the classification and may be determined empirically or by a machine learning process.

[0082] The texture circuit 802 is a circuit that determines texture values ​​that represent the likelihood that the content categories identified by the content map 504 have been corrected based on texture information. In one or more embodiments, the texture values ​​are determined by applying one or more edge detection operations to the input image. For example, an edge detection method such as a Sobel filter or a high pass frequency filter is applied to the luminance input image to obtain edge values ​​at pixel locations of the input image. After the edge values ​​are determined, a texture value for a grid point may be determined by applying the edge values ​​to a texture model. The texture circuit 802 may store a number of different texture models that correspond to different content categories. Examples of texture models include a leaf texture model, a sky texture model, a grass texture model, and a skin texture model. An exemplary leaf texture model is described below with respect to FIG. 9.

[0083] The chroma circuit 804 is a circuit that determines a chroma value that represents the likelihood that the content category identified by the content map 504 is corrected based on chroma information. The chroma value is based on the color information (e.g., Cb and Cr values) of the image. The chroma circuit 804 may store different chroma models for different content categories. Examples of chroma models include a leaf chroma model, a sky chroma model, a grass chroma model, and a skin chroma model. The chroma models may be determined manually or by machine learning techniques. An exemplary sky chroma model is described below with respect to FIG. 10.

[0084] The content modification circuit 810 receives the content coefficients from the content coefficient circuit 806 and the delta Y values ​​from the image sharpener 702. The content modification circuit 810 applies the content coefficients to the delta Y values ​​to generate delta Y' values. For example, if the content coefficient of a pixel is above a predetermined threshold (e.g., 0), the content modification circuit 810 performs a sharpening operation, such as multiplying the content coefficient of the pixel by the delta Y value of the pixel. If the content coefficient of a pixel is below the predetermined threshold, the content modification circuit 810 may perform a smoothing operation by blending the delta Y' based on the content coefficient. For example, alpha blending is performed according to the following formula: Y'=(1-alpha) * Y+Alpha * (Y-Delta Y) (3) and, Y'=Y+Delta Y' (4) therefore, Delta Y' = -(Alpha) * (Delta Y) (5). where alpha = |Q| * Let |Q| be the scale and alpha be a value between 0 and 1. Let |Q| be a predefined positive constant. * Note that if |Q| is large enough such that scale>1, alpha is clipped to 1.

[0085] FIG. 9 is a plot showing a texture model, according to one embodiment. If a pixel (or grid point, if a grid is used) is classified as a "leaf" by the content map, for example, a texture value may be determined by applying the edge value of the pixel (or grid point, if a grid is used) to the model of FIG. 9. The x-axis represents the input edge value, and the y-axis represents the output texture value. The texture values ​​range from 0 to 1. If the edge value of the pixel is above a high threshold, the texture value is 1, and if the edge value is below a low threshold, the texture value is 0. The texture value increases linearly from 0 to 1 for edge values ​​between the low and high thresholds. The values ​​of the low and high thresholds may be determined empirically. For example, the thresholds are set according to typical edge values ​​for leaves. Thus, an edge value above a high threshold may indicate that the pixel or grid point is in an area with high texture corresponding to leaves. Similarly, an edge value below a low threshold may indicate that the pixel or grid point is in an area with flat texture not corresponding to leaves. For example, according to Equation 2, a high texture value indicates a high likelihood that the pixel content is a leaf (assuming C2 is positive).

[0086] For different categories, the corresponding texture models may be represented by different texture parameters (e.g., different low thresholds, different high thresholds, and / or different gradients, and / or flipping 1 and 0). For the category of "grass", the low and high thresholds may be higher than those of the category of "leaves". Some categories are more likely to be correct when the texture is flat than when the texture is complex. For such categories, the texture values ​​of 1 and 0 may be flipped. For example, the texture model of "sky" may have a value of 1 when the edge value is lower (e.g., below the low threshold) and a value of 0 when the edge value is higher (e.g., above the high threshold). That is, pixels that are in regions of the input image where the texture is flat are more likely to indicate "sky". In some embodiments, instead of flipping the values ​​of 1 and 0, the predetermined constant in Equation 2 varies based on the content category. For example, for categories such as "skin" or "sky", C2 may be negative such that a high texture value indicates that the pixel's content is less likely to be "skin" or "sky".

[0087] FIG. 10 is a plot showing a chroma model according to one embodiment. The chroma model of FIG. 10 may represent a model of "sky". If a pixel or grid point is classified as "sky" by the content map, the chroma circuit 804 determines whether the combination of Cb and Cr values ​​for the pixel or grid point falls within one of areas 1010, 1020, and 1030. The x-axis represents the input Cb value, and the y-axis represents the input Cr value. An ellipse is located in the upper right corner of the plot representing the color range of "sky". If the Cb / Cr value of the pixel or grid point is within the inner ellipse in area 1010, the chroma value is 1, which may indicate a high likelihood that the pixel or grid point corresponds to "sky" (e.g., according to Equation 2 where C3 is positive). If the Cb / Cr value is outside the outer ellipse in area 1030, the chroma value is 0, which indicates a low likelihood that the pixel or grid point corresponds to "sky" (e.g., according to Equation 2 where C3 is positive). If the Cb / Cr values ​​are between the inner and outer ellipses in area 1020, the chroma values ​​may be between 0 and 1 (e.g., the chroma values ​​increase as the distance from the edge of the inner ellipse increases). The location and size of the ellipse may be determined empirically or by statistical models. In some embodiments, other shapes such as squares, triangles, or circles may be used instead of ellipses.

[0088] For different categories, the corresponding chroma models may be represented by different chroma parameters (e.g., center, radius, angle, tilt of the ellipse, and ratio of outer / inner ellipse). For example, the chroma model of "leaf" is usually green, and therefore the chroma parameters of the "leaf" category cover Cb / Cr values ​​corresponding to green, while the chroma parameters of the "sky" category cover Cb / Cr values ​​corresponding to blue. In another example, the chroma parameters are selected according to colors that are not generally associated with the category. For example, "leaf" does not generally include blue. Thus, the chroma model may be configured to output low values ​​when the Cb / Cr values ​​correspond to blue.

[0089] Example of Interpolation Using Lattice Points FIG. 11 illustrates a method for upsampling a content map, according to one embodiment. If the content map has a lower resolution than the input image, there is not a one-to-one correspondence between pixels in the image and pixels in the content map. Thus, upscaling of the content map can be performed. One way to perform such upscaling is by use of a grid with multiple grid points superimposed on the input image 1102. The grid points may be sparser than the pixels in the input image 1102. In such a case, the content coefficients of the grid points can be used to determine the content coefficients of the higher resolution input image 1102.

[0090] Taking an example where a grid overlaid on an input image 1102 has grid points 1-4, and the input image 1102 includes a pixel 1104 that is located between grid points 1-4, the content coefficient Q(pixel) for pixel 1104 may be determined by performing bilinear interpolation on the content coefficients Q(1)-Q(4) associated with grid points 1-4, taking into account the spatial distance from the grid point to pixel 1104.

[0091] In one or more embodiments, the content coefficients Q(1)-Q(4) for determining Q(pixel) for pixel 1104 are determined using texture and chroma parameters for pixel 1104 of input image 1102. The categories of lattice points 1-4 indicated by the content map are used, but the likelihood values ​​for these categories (described above with reference to equation (2)) are determined using the edge and Cb / Cr values ​​of pixel 1104 rather than the edge and Cb / Cr values ​​of the lattice points. For example, if lattice point 1 is classified as "skin", then the texture of pixel 1104 is applied to a texture model having texture parameters corresponding to "skin", and the Cb / Cr values ​​of pixel 1104 are applied to a chroma model having chroma parameters corresponding to "skin" to obtain Q(1) according to equation (1). Similarly, if grid point 2 is classified as "leaf", then the edge value of pixel 1104 is applied to a texture model having texture parameters corresponding to "leaf", and the Cb / Cr values ​​of pixel 1104 are applied to a chroma model having chroma parameters corresponding to "leaf" to obtain Q(2) according to equation (1). A similar process is repeated for grid points 3 and 4, and then Q(pixel) is obtained by bilinear interpolation.

[0092] In another embodiment, the content coefficients Q(1) to Q(4) are obtained by using the texture values ​​and Cb / Cr values ​​of the grid points instead of the texture values ​​and Cb / Cr values ​​of the pixels.

[0093] Exemplary Methods for Content-Based Image Sharpening 12 is a flow chart illustrating a method for sharpening one or more pixels of an image based on content within a segment of the image, according to one embodiment. The steps of the method may be performed in a different order, and the method may include different, additional, or fewer steps.

[0094] Luminance pixel values ​​of an input image are received (1202). A content map is received (1204). The content map identifies categories of content within a segment of the image. The content map may be generated by a neural processor circuit that performs at least one machine learning operation on a version of the image to generate the content map. Categories identified by the content map may include skin, leaves, grass, or sky. In some embodiments, the content map is a heat map that indicates an amount of sharpening to be applied to pixels of the image.

[0095] A content coefficient associated with a pixel in the image is determined (1206). The content coefficient is determined according to a category of the identified content in the input image and at least one of a texture value of the pixel in the image or a chroma value of the pixel in the image. The texture value indicates the likelihood of a category of content associated with the pixel based on the texture in the image. The chroma value indicates the likelihood of a category of content associated with the pixel based on color information of the image.

[0096] In some embodiments, the content map is downscaled relative to the input image. In these embodiments, the content coefficients may be determined by upsampling the content map. The content map may be upsampled by taking content coefficients of lattice points overlaid on the content map and then interpolating the content coefficients of the lattice points to obtain content coefficients of pixels in the input image.

[0097] A sharpened version of the luma pixel value for the pixel is generated (1208) by applying at least the content coefficient to a version of the luma pixel value for the pixel. The version of the luma pixel value may be generated by a bilateral filter or a high-pass filter. In some embodiments, the content coefficient is applied to the version of the luma pixel value by multiplying the content coefficient with the version of the luma pixel value in response to the content coefficient being above a threshold. If the content coefficient is below the threshold, the content modification circuit applies a version of the luma pixel value multiplied by a negative parameter.

[0098] The teachings described herein relate to generating a content coefficient for each pixel in an image. The content coefficient is based on the content in the image identified via a content map. The teachings described herein are described in the context of image sharpening, but this is for convenience. The teachings described herein can also be applied to other image processing processes, such as noise reduction, tone mapping, and white balance processes. For example, for noise reduction, the content coefficient can be applied (e.g., multiplied) to the noise standard deviation.

[0099] While particular embodiments and applications have been illustrated and described, it should be understood that the invention is not limited to the precise construction and components disclosed herein, and various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope of the disclosure.

Claims

1. a content image processing circuit configured to receive a delta version of a luminance pixel value of a pixel of an image and a content map identifying a category of content within a segment of the image, the delta version of the luminance pixel value of the pixel being generated by performing contrast enhancement on luminance information of the luminance pixel value of the pixel; The content image processing circuit, determining a content coefficient associated with the pixel of the image according to the identified category of content in the image and at least one of a texture value of the pixel or a chroma value of the pixel, the texture value indicating a likelihood of a category of content associated with the pixel based on a texture in the image, and the chroma value indicating a likelihood of a category of content associated with the pixel based on color information of the image; generating a processed version of the luminance pixel value of the pixel by applying the content coefficient to at least the delta version of the luminance pixel value of the pixel; further comprising: A device for image processing.

2. The apparatus of claim 1 , wherein the processed versions of the luminance pixel values ​​are denoised versions of the luminance pixel values.

3. The apparatus of claim 1 , wherein the delta version of the luminance pixel value of the pixel comprises high frequency components of the image.

4. The apparatus of claim 1 , wherein the processed versions of the luminance pixel values ​​are white balanced versions of the luminance pixel values.

5. The apparatus of claim 1 , wherein the delta version of the luma pixel value of the pixel is generated by a bilateral filter or a high pass filter applied to the luma information of the luma pixel value of the pixel.

6. 2. The apparatus of claim 1, wherein the content image processing circuitry is configured to apply the content coefficients to the delta versions of the luma pixel values ​​by: (i) performing a first image processing operation in response to the content coefficient exceeding a threshold; and (ii) performing a second, different image processing operation in response to the content coefficient being below the threshold.

7. The apparatus of claim 6 , wherein the first and second image processing operations are noise adjustment operations.

8. The apparatus of claim 6 , wherein the first and second image processing operations are tone mapping operations.

9. The apparatus of claim 6 , wherein the first and second image processing operations are white balance operations.

10. receiving a delta version of a luminance pixel value of a pixel of an image, the delta version of the luminance pixel value of the pixel being generated by performing contrast enhancement on luminance information of the luminance pixel value of the pixel; receiving a content map that identifies categories of content within a segment of the image; determining a content coefficient associated with the pixel of the image according to the identified category of content in the image and at least one of a texture value of the pixel or a chroma value of the pixel, the texture value indicating a likelihood of a category of content associated with the pixel based on a texture in the image, and the chroma value indicating a likelihood of a category of content associated with the pixel based on color information of the image; generating a processed version of the luma pixel value of the pixel by applying the content coefficient to at least the delta version of the luma pixel value of the pixel; A method comprising:

11. The method of claim 10 , wherein the processed versions of the luminance pixel values ​​are denoised versions of the luminance pixel values.

12. The method of claim 10 , wherein the delta version of the luminance pixel value of the pixel comprises high frequency components of the image.

13. The method of claim 10 , wherein the processed versions of the luminance pixel values ​​are white balanced versions of the luminance pixel values.

14. The method of claim 10 , wherein the delta version of the luminance pixel value of the pixel is generated by a bilateral filter or a high-pass filter applied to the luminance information of the luminance pixel value of the pixel.

15. The method of claim 10, wherein applying the content coefficient to the delta version of the luminance pixel value comprises: (i) performing a first image processing operation in response to the content coefficient exceeding a threshold; and (ii) performing a second, different image processing operation in response to the content coefficient being below the threshold.

16. a neural processor circuit configured to run a machine learning algorithm on an image to generate a content map that identifies categories of content within a segment of the image; a content image processing circuit coupled to the neural processor circuit and configured to receive a delta version of a luminance pixel value of a pixel of the image and the content map, the delta version of the luminance pixel value of the pixel being generated by performing contrast enhancement on luminance information of the luminance pixel value of the pixel; Including, The content image processing circuit, determining a content coefficient associated with the pixel of the image according to the identified category of content in the image and at least one of a texture value of the pixel or a chroma value of the pixel, the texture value indicating a likelihood of a category of content associated with the pixel based on a texture in the image, and the chroma value indicating a likelihood of a category of content associated with the pixel based on color information of the image; receiving the content coefficients and generating a processed version of the luma pixel value of the pixel by at least applying the content coefficients to the delta version of the luma pixel value of the pixel; further comprising: Electronic devices.

17. The electronic device of claim 16 , wherein the processed versions of the luminance pixel values ​​are denoised versions of the luminance pixel values.

18. The electronic device of claim 16 , wherein the delta version of the luminance pixel value of the pixel comprises high frequency components of the image.

19. 17. The electronic device of claim 16, wherein the delta version of the luminance pixel value of the pixel is generated by a bilateral filter or a high pass filter applied to the luminance information of the luminance pixel value of the pixel.

20. 17. The electronic device of claim 16, wherein the content image processing circuitry is configured to apply the content coefficients to the delta versions of the luma pixel values ​​by (i) performing a first image processing operation in response to the content coefficient exceeding a threshold, and (ii) performing a second, different image processing operation in response to the content coefficient being below the threshold.

Images