Systems and methods for resolution extraction and recovery for high-resolution image processing

Abstract

Systems and methods related to obtaining, by an image signal processor (ISP), an input image having a first resolution; extracting, by a first portion of the ISP, high resolution information from the input image to generate a modified input image having a second resolution that is smaller than the first resolution; outputting, for further image processing by the ISP, the modified input image; storing the extracted high resolution information in a resolution extraction buffer; receiving, by a second portion of the ISP, a processed version of the modified input image after processing by the ISP; retrieving the extracted high resolution information from the resolution extraction buffer; performing, by the ISP, resolution recovery by adding the extracted high resolution information to the processed modified input image to generate an output image having the first resolution; and providing, by the ISP and for image post-processing, the output image.

Description

SYSTEMS AND METHODS FOR RESOLUTION EXTRACTION AND RECOVERY FOR HIGH-RESOLUTION IMAGE PROCESSINGBACKGROUND[1] High-resolution processing may be performed in image processing pipelines in camera systems. An image signal processor may downscale an image at the beginning of image processing, and subsequently upscale the image after processing.SUMMARY[2] High-resolution processing (e.g. 8K video frames) can result in increased power consumption. For example, reading and writing an 8K frame buffer can consume power in a few hundreds of megawatts (mW) at 30 frame per second (fps). To overcome this, an image signal processor (ISP) may scale down an image early in the image processing pipeline and then performs an upscaling after intermediate image processing is performed. Such a downscaling and upscaling can result in a significant reduction in detail and resolution of the image.[3] This application generally relates to an ISP that performs high-resolution processing using a high-frequency extraction component and a resolution recovery component, with minimal processing power. This can result in saving dynamic random access memory (DRAM) bandwidth (BW), and chip area reduction.[4] In a first example embodiment, a method may include obtaining, by a first portion of an image signal processor (ISP), an input image from an image sensor, the input image having a first resolution. The method may include extracting, by the first portion of the ISP, high resolution information from the input image having the first resolution to generate a modified input image having a second resolution that is smaller than the first resolution. The method may also include outputting, by the first portion of the ISP and for further image processing by the ISP, the modified input image having the second resolution. The method may additionally include storing, by the first portion of the ISP, the extracted high resolution information in a resolution extraction buffer. The method may also include receiving, by a second portion of the ISP, a processed version of the modified input image after processing by the ISP. The method may further include retrieving, by the second portion of the ISP, the extracted high resolution information from the resolution extraction buffer. The method may also include performing, by the second portion of the ISP, resolution recovery by adding the extracted highresolution information to the processed version of the modified input image to generate an output image having the first resolution. The method may additionally include providing, by the second portion of the ISP and for image post-processing, the output image having the first resolution.[5] In a second example embodiments, a system may include an image reception circuitry configured to obtain an input image from an image sensor, the input image having a first resolution. The system may also include a resolution extraction circuitry situated at a first portion of an image signal processor (ISP), the resolution extraction circuitry configured to extract high resolution information from the input image having the first resolution to generate a modified input image having a second resolution that is smaller than the first resolution, output, for further image processing by the ISP, the modified input image having the second resolution, and store the extracted high resolution information in a resolution extraction buffer. The system may further include a resolution recovery circuitry situated at a second portion of the ISP, the resolution recovery circuitry configured to receive a processed version of the modified input image after processing by the ISP, retrieve the extracted high resolution information from the resolution extraction buffer, perform resolution recovery by adding the extracted high resolution information to the processed version of the modified input image to generate an output image having the first resolution, and provide, for post-processing, the output image having the first resolution.[6] In a third example embodiment, a computing device may include a processor and a non- transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations in accordance with the first example embodiment and / or the second example embodiment.[7] These, as well as other embodiments, aspects, advantages, and alternatives, will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying drawings. Further, this summary and other descriptions and figures provided herein are intended to illustrate embodiments by way of example only and, as such, that numerous variations are possible. For instance, structural elements and process steps can be rearranged, combined, distributed, eliminated, or otherwise changed, while remaining within the scope of the embodiments as claimed.BRIEF DESCRIPTION OF THE FIGURES[8] FIG. 1 illustrates a computing device, in accordance with example embodiments.[9] FIG. 2 illustrates a computing system, in accordance with example embodiments.

[0010] FIG. 3 illustrates an example of image processing without resolution recovery, in accordance with example embodiments.

[0011] FIG. 4 illustrates an example of image processing with resolution recovery, in accordance with example embodiments.

[0012] FIG. 5 illustrates high-frequency extraction, in accordance with example embodiments.

[0013] FIG. 6 illustrates another example of image processing with resolution recovery, in accordance with example embodiments.

[0014] FIG. 7 illustrates resolution recovery, in accordance with example embodiments.

[0015] FIG. 8 illustrates example images based on image processing with and without resolution recovery, in accordance with example embodiments.

[0016] FIG. 9 illustrates an example of image processing with a non-Bayer processing block, in accordance with example embodiments.

[0017] FIG. 10 illustrates an example color filter array (CFA) pattern and frequency spectrum for a Bayer pattern and a quad Bayer pattern, in accordance with example embodiments.

[0018] FIG. 11 A illustrates an example binning applied to a quad Bayer pattern, in accordance with example embodiments.

[0019] FIG. 11B illustrates example chroma alias artifacts for a quad Bayer pattern after binning and after remosaicing, in accordance with example embodiments.

[0020] FIG. 12 illustrates chroma noise artifacts for a quad Bayer pattern after binning and after remosaicing, in accordance with example embodiments.

[0021] FIG. 13 illustrates an example of image processing with binning and luma high- frequency extraction, in accordance with example embodiments.

[0022] FIG. 14 is a flowchart of a method, in accordance with example embodiments.DETAILED DESCRIPTION

[0023] Example methods, devices, and systems are described herein. It should be understood that the words “example” and “exemplary” are used herein to mean “serving as an example, instance, or illustration.” Any embodiment or feature described herein as being an “example,” “exemplary,” and / or “illustrative” is not necessarily to be construed as preferred or advantageous over other embodiments or features unless stated as such. Thus, other embodiments can be utilized and other changes can be made without departing from the scope of the subject matter presented herein.

[0024] Accordingly, the example embodiments described herein are not meant to be limiting. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0025] Further, unless context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be generally viewed as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are necessary for each embodiment.

[0026] Additionally, any enumeration of elements, blocks, or steps in this specification or the claims is for purposes of clarity. Thus, such enumeration should not be interpreted to require or imply that these elements, blocks, or steps adhere to a particular arrangement or are carried out in a particular order. Unless otherwise noted, figures are not drawn to scale.Overview

[0027] Processing high resolution image data can involve higher power consumption and may use image buffers configured to process the high resolution image data. Although high- frequency (HF) extraction may be performed using software-based implementations, any potential power savings resulting from processing low-resolution images based on a softwarebased HF extraction may be lost in the software implementation itself. Also, for example, software-based HF extraction may involve providing the software components access to the internal ISP buffers and this may add to power consumption. Accordingly, a hardware implementation may be more cost-effective in some instances.

[0028] Some power saving approaches involve downscaling the image early in the image processing pipeline and perform upscaling at a later time. However, such approaches may result in a significant reduction in the detail and resolution of the image. Resolution enhancement approaches may also be applied during post-processing. However, such approaches use significant amounts of computational resources for certain inputs (e.g., higher resolution inputs such as 4K).

[0029] As described herein, a high-frequency extraction component may be introduced early in the image processing pipeline to extract high-frequency information from the incoming sensor data. The extracted high-frequency data may be stored in a high-frequency buffer and can be re-introduced in a later portion of the image processing pipeline. In some aspects, luma extraction may extract luma high-frequency information and store that in a luma buffer to bere-introduced in a later portion of the image processing pipeline. Once the high-frequency data and / or luma information is extracted, the image data is reduced to a lower resolution and this lower resolution image data may be provided to the remainder of the image processing pipeline. Such an approach enables data flows, storing, and processing at a lower resolution, thereby resulting in power savings and memory resource savings. Upon processing of the lower resolution image data, a resolution recovery component in a later portion of the image processing pipeline may re-introduce the high-frequency and / or luma information. This results in a high resolution image and also reduces image degradations.Example Computing Devices and Systems

[0030] FIG. 1 illustrates an example computing device 100, in accordance with example embodiments. Computing device 100 is shown in the form factor of a mobile phone. However, computing device 100 may be alternatively implemented as a laptop computer, a tablet computer, and / or a wearable computing device, among other possibilities. Computing device 100 may include various elements, such as body 102, display 106, and buttons 108 and 110. Computing device 100 may further include one or more cameras, such as front-facing camera 104 and rear-facing camera 112.

[0031] Front-facing camera 104 may be positioned on a side of body 102 typically facing a user while in operation (e.g., on the same side as display 106). Rear-facing camera 112 may be positioned on a side of body 102 opposite front-facing camera 104. Referring to the cameras as front and rear facing is arbitrary, and computing device 100 may include multiple cameras positioned on various sides of body 102.

[0032] Display 106 could represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light emitting diode (OLED) display, or any other type of display known in the art. In some examples, display 106 may display a digital representation of the current image being captured by front-facing camera 104 and / or rear-facing camera 112, an image that could be captured by one or more of these cameras, an image that was recently captured by one or more of these cameras, and / or a modified version of one or more of these images. Thus, display 106 may serve as a viewfinder for the cameras. Display 106 may also support touchscreen functions that may be able to adjust the settings and / or configuration of one or more aspects of computing device 100.

[0033] Front-facing camera 104 may include an image sensor and associated optical elements such as lenses. Front-facing camera 104 may offer zoom capabilities or could have a fixed focallength. In other examples, interchangeable lenses could be used with front-facing camera 104. Front-facing camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. Front-facing camera 104 also could be configured to capture still images, video images, or both. Further, front-facing camera 104 could represent, for example, a monoscopic, stereoscopic, or multiscopic camera. Rear-facing camera 112 may be similarly or differently arranged. Additionally, one or more of front-facing camera 104 and / or rear-facing camera 112 may be an array of one or more cameras.

[0034] Computing device 100 could be configured to use display 106 and front-facing camera 104 and / or rear-facing camera 112 to capture images of a target object. The captured images could be a plurality of still images or a video stream. The image capture could be triggered by activating button 108, pressing a softkey on display 106, or by some other mechanism. Depending upon the implementation, the images could be captured automatically at a specific time interval, for example, upon pressing button 108, upon appropriate lighting conditions of the target object, upon moving computing device 100 a predetermined distance, or according to a predetermined capture schedule.

[0035] FIG. 2 is a simplified block diagram showing some of the components of an example computing system 200. By way of example and without limitation, computing system 200 may be a cellular mobile telephone (e.g., a smartphone), a computer (such as a desktop, notebook, tablet, server, or handheld computer), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a gaming console, a robotic device, a vehicle, or some other type of device. Computing system 200 may represent, for example, aspects of computing device 100.

[0036] As shown in FIG. 2, computing system 200 may include communication interface 202, user interface 204, processor 206, data storage 208, and camera components 224, all of which may be communicatively linked together by a system bus, network, or other connection mechanism 210. Computing system 200 may be equipped with at least some image capture and / or image processing capabilities. It should be understood that computing system 200 may represent a physical image processing system, a particular physical hardware platform on which an image sensing and / or processing application operates in software, or other combinations of hardware and software that are configured to carry out image capture and / or processing functions.

[0037] Communication interface 202 may allow computing system 200 to communicate, using analog or digital modulation, with other devices, access networks, and / or transport networks.Thus, communication interface 202 may facilitate circuit-switched and / or packet-switched communication, such as plain old telephone service (POTS) communication and / or Internet protocol (IP) or other packetized communication. For instance, communication interface 202 may include a chipset and antenna arranged for wireless communication with a radio access network or an access point. Also, communication interface 202 may take the form of or include a wireline interface, such as an Ethernet, Universal Serial Bus (USB), or High -Definition Multimedia Interface (HDMI) port, among other possibilities. Communication interface 202 may also take the form of or include a wireless interface, such as a Wi-Fi, BLUETOOTH®, global positioning system (GPS), or wide-area wireless interface (e.g, WiMAX or 3GPP Long- Term Evolution (LTE)), among other possibilities. However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over communication interface 202. Furthermore, communication interface 202 may comprise multiple physical communication interfaces (e.g, a Wi-Fi interface, a BLUETOOTH® interface, and a wide-area wireless interface).

[0038] User interface 204 may function to allow computing system 200 to interact with a human or non-human user, such as to receive input from a user and to provide output to the user. Thus, user interface 204 may include input components such as a keypad, keyboard, touch-sensitive panel, computer mouse, trackball, joystick, microphone, and so on. User interface 204 may also include one or more output components such as a display screen, which, for example, may be combined with a touch-sensitive panel. The display screen may be based on CRT, LCD, LED, and / or OLED technologies, or other technologies now known or later developed. User interface 204 may also be configured to generate audible output(s), via a speaker, speaker jack, audio output port, audio output device, earphones, and / or other similar devices. User interface 204 may also be configured to receive and / or capture audible utterance(s), noise(s), and / or signal(s) by way of a microphone and / or other similar devices.

[0039] In some examples, user interface 204 may include a display that serves as a viewfinder for still camera and / or video camera functions supported by computing system 200. Additionally, user interface 204 may include one or more buttons, switches, knobs, and / or dials that facilitate the configuration and focusing of a camera function and the capturing of images. It may be possible that some or all of these buttons, switches, knobs, and / or dials are implemented by way of a touch-sensitive panel.

[0040] Processor 206 may comprise one or more general purpose processors - e.g., microprocessors - and / or one or more special purpose processors - e.g., digital signalprocessors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, application-specific integrated circuits (ASICs), and / or tensor processing units (TPUs). In some instances, special purpose processors may be capable of image processing, image alignment, and merging images, among other possibilities. Data storage 208 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integrated in whole or in part with processor 206. Data storage 208 may include removable and / or non-removable components.

[0041] Processor 206 may be capable of executing program instructions 218 (e.g., compiled or non-compiled program logic and / or machine code) stored in data storage 208 to carry out the various functions described herein. Therefore, data storage 208 may include a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by computing system 200, cause computing system 200 to carry out any of the methods, processes, or operations disclosed in this specification and / or the accompanying drawings. The execution of program instructions 218 by processor 206 may result in processor 206 using data 212.

[0042] By way of example, program instructions 218 may include an operating system 222 (e.g., an operating system kernel, device driver(s), and / or other modules) and one or more application programs 220 (e.g., camera functions, address book, email, web browsing, social networking, audio-to-text functions, text translation functions, and / or gaming applications) installed on computing system 200. Similarly, data 212 may include operating system data 216 and application data 214. Operating system data 216 may be accessible primarily to operating system 222, and application data 214 may be accessible primarily to one or more of application programs 220. Application data 214 may be arranged in a file system that is visible to or hidden from a user of computing system 200.

[0043] Application programs 220 may communicate with operating system 222 through one or more application programming interfaces (APIs). These APIs may facilitate, for instance, application programs 220 reading and / or writing application data 214, transmitting or receiving information via communication interface 202, receiving and / or displaying information on user interface 204, and so on.

[0044] In some cases, application programs 220 may be referred to as “apps” for short. Additionally, application programs 220 may be downloadable to computing system 200 through one or more online application stores or application markets. However, applicationprograms can also be installed on computing system 200 in other ways, such as via a web browser or through a physical interface (e.g., a USB port) on computing system 200.

[0045] Camera components 224 may include, but are not limited to, an aperture, shutter, recording surface (e.g., photographic film and / or an image sensor), lens, shutter button, infrared projectors, and / or visible-light projectors. Camera components 224 may include components configured for capturing of images in the visible-light spectrum (e.g., electromagnetic radiation having a wavelength of 380 - 700 nanometers) and / or components configured for capturing of images in the infrared light spectrum (e.g., electromagnetic radiation having a wavelength of 701 nanometers - 1 millimeter), among other possibilities. Camera components 224 may be controlled at least in part by software executed by processor 206.Example Image Processing Systems

[0046] FIG. 3 illustrates an example of image processing 300 without resolution recovery, in accordance with example embodiments. Some embodiments involve obtaining, by a first portion of an image signal processor (ISP), an input image from an image sensor, the input image having a first resolution. For example, input image may be received as raw data from an image sensor 305. The image sensor is illustrated to be based on a Bayer pattern 305A. However, image sensors configured in a non-Bayer pattern may be handled by converting the non-Bayer data to Bayer data (e.g., by remosaicing, binning, etc.). The input image may have a first resolution (e.g., 16K, 14K, 8K, 4K, etc.).

[0047] The image sensor 305 may include a plurality of light-sensing pixels that measure an intensity of light incident thereon and thereby collectively capture an image of an environment. The light-sensing pixels may be arranged in arrays, and may be grouped by position into pixel sensor groups. For example, an array of pixel sensors may be grouped into pixel sensor groups, where each pixel sensor group includes an array of pixel sensors. Such pixel sensor groups may be further grouped to define regions of interest (ROIs). Image processing circuitry groups may be configured to each receive pixel information from a corresponding pixel sensor group and further configured to perform image processing operations on the pixel information to provide processed pixel information during operation of the image sensor.

[0048] In the case of image processing, the input is typically either singular or a set of three channel images consisting of red, green, and blue (RGB) color channels. Such color images are commonly captured with an array of light sensors covered with different color filters, called a color filter array (CFA), which are generally arranged as a repeated mosaic. Examples ofcolor filters can be, for example, a Bayer filter, a modified Bayer filter such as RGBE where a green filter is modified to an “emerald” filter, a red-yellow-yellow-blue (RYYB) filter, a cyanyellow-yellow-magenta (CYYM) filter, a cyan-yellow-green-magenta (CYGM) filter, various modifications of the Bayer filter (e.g., RGBW where a green filter is modified to a “white” filter, a Quad Bayer filter (comprising 4x blue, 4x red, and 8x green filters), RYYB Quad Bayer (comprising 4x blue, 4x red, and 8x yellow filters), nonacell (comprising 9x blue, 9x red, and 18x green filters), RCCC filter (comprising a monochrome sensor with a red channel), RCCB filter (where the green pixels are clear), and others. For example, the color filter can also correspond to filters used in multispectral sensors. For low latency and memory bound applications it is sometimes desirable to process the raw color mosaic image as opposed to some RGB equivalent that was reconstructed from the raw image.

[0049] As illustrated, in the Bayer pattern 305A each 2x2 array of pixels has a total of two green pixels, one red pixel, and one blue pixel. These are arranged (beginning at the top left corner of the illustrated Bayer pattern 305 A) as a green pixel (G) and a red pixel (R) in the top row, followed by a blue pixel (B) and a green pixel (G) in the bottom row. Accordingly, odd numbered rows are arranged as “GRGR...” and even numbered rows are arranged as “BGBG...”

[0050] High-resolution (c.g, 8K) raw image data from image sensor 305 may be provided to an image signal processor (ISP). In some embodiments, the ISP may include a front-end (ISP- FE) 310 and a back-end (ISP -BE) 320. Generally, ISP-FE 310 is in direct communication with the image sensor 305 via a sensor interface. Accordingly, raw image data is streamed into ISP- FE 310 in real time. For example, during video capture, raw image data in one or more frames of the captured video may be streamed into ISP-FE 310 in real time. Accordingly, ISP-FE 310 may be generally configured for real-time image processing, such as sensor related access, sensor correction, and so forth. For example, the output from image sensor 305 may have a defective pixel that may need to be corrected. Also, for example, raw image data may be indicative of strands and / or image comers may have darker illumination (e.g., shading artifacts) that may need to be corrected, and / or mitigated. Accordingly, ISP-FE 310 may be configured to correct such defects. However, processing of a large number of frames in real time or near real time may result in issues in bottlenecks in the image processing pipeline, utilize more resources, both in terms of power and compute resources. Accordingly, there is a need to make image processing more efficient while using fewer resources to achieve faster speeds, reliability, and image quality.

[0051] As ISP-FE 310 is directly connected to image sensor 305, there is a need for the processing to be performed in real-time and to match an output from the sensor. Some embodiments include memory circuitry configured as an intermediate buffer 315 between the front-end of the ISP (e.g., ISP-FE 310) and a back-end of the ISP (e.g., ISP-BE 320). The intermediate buffer 315 may be configured to store a modified input image processed by ISP- FE 310. Depending on a placement of a resolution extraction circuitry, the intermediate buffer may be configured to store image data having a same resolution as the input image data, or image data having a lower resolution that the raw image data. For example, intermediate buffer 315 may separate ISP-FE 310 from ISP-BE 320. Accordingly, in some embodiments, more detailed image processing may be performed at ISP-BE 320. For example, when the input image is a Bayer image, the output fine color image viewable by a user is generated by ISP- BE 320. Accordingly, operations such as high dynamic range processing, temporal noise reduction, upsampling, etc. may be performed in ISP-BE 320. Generally speaking, ISP-FE 310 and ISP-BE 320 may reside on the same piece of silicon.

[0052] The high-resolution raw image data may be stored in intermediate buffer 315. Generally, in existing image processing pipelines, intermediate buffer 315 is configured to process raw image data of the same resolution as input image data from image sensor 305. For example, when the input image data is 8K, intermediate buffer 315 is configured for 8K storage and processing. Also, for example, when the input image data is 16K, intermediate buffer 315 is configured for 16K storage and processing.

[0053] In some embodiments, high dynamic range (HDR) processing may be performed in ISP-BE 320. For example, a HDR processor may access the high-resolution raw image data from intermediate buffer 315 for further processing. In some embodiments, HDR processing may be performed by a Bayer pattern processor, and temporal noise reduction (TNR) may be performed on an output of the Bayer pattern processor. In some embodiments, ISP-BE 320 may be configured to perform such functions iteratively to reduce noise. For example, intermediate processed images may be stored in a frame buffer 325 by one or more components and / or buffers of ISP-BE 320 for access by the same or other components and / or buffers of ISP-BE 320. Accordingly, in processing high-resolution image data, the components and / or buffers of ISP-BE 320 may also consume higher power resources, and memory resources.

[0054] In some embodiments, a low-frequency (LF) processor, such as YUV-LF may access the high-resolution image data. The letter “Y” generally refers to luma or brightness, the letter “U” generally refers to a blue projection, and the letter “V” generally refers to a red projection.Also, for example, a high-frequency (HF) processor, such as YUV-HF, may access the high- resolution image data. Subsequently, ISP-BE 320 may output a processed high-resolution image data and save it in an output buffer 355. One or more post-processing components of a camera system may access the processed high-resolution image data from the output buffer 355.

[0055] As illustrated in this high level image processing pipeline process, the raw data is processed and transmitted in high-resolution, resulting in higher power consumption. Also, in many instances, even though the processing is performed on high-resolution image data, the final output (e.g., an image viewed by a user) may be of a lower resolution. As such, higher power resources may be consumed that does not impact a quality of the output image.

[0056] FIG. 4 illustrates an example of image processing 400 with resolution recovery 400, in accordance with example embodiments. For example, an ISP can perform high-resolution (e.g. 16K, 8K frames) video processing by extracting the high-frequency data and implementing a resolution recovery process that adds the extracted high-frequency data at a later portion of the ISP pipeline. At a high level, such a process allows ISP processing to be performed on low- resolution data, while buffers are also configured for such low-resolution data. This enables a reduction in power consumption by the ISP pipeline.

[0057] Although high-frequency (HF) extraction may be performed using software-based implementations, any potential power savings resulting from processing low-resolution images based on a software-based HF extraction may be lost in the software implementation itself. Also, for example, software-based HF extraction may involve providing the software components access to the internal ISP buffers (e.g., intermediate buffer 315, frame buffer 325, etc.) and this may add to power consumption. Accordingly, a hardware implementation may be more cost-effective in some instances.

[0058] FIG. 4 shares one or more aspects in common with FIG. 3. Similar reference numerals are used to indicate components that are shared by FIGs. 3 and 4. For example, high-resolution image data may be stored in intermediate buffer 315 after initial processing by ISP-FE 310. As previously described, ISP-BE 320 may access the initially processed high-resolution image data from intermediate buffer 315. Some embodiments involve extracting, by the first portion of the ISP, high resolution information from the input image having the first resolution to generate a modified input image having a second resolution that is smaller than the first resolution. In some embodiments, a high-frequency (HF) extraction may be performed by a hardware HF extraction 335A component within ISP-BE 320. In some embodiments, suchhigh-frequency extraction may be performed with a lower level of precision. In image processing, the term “frequency” generally refers to a rate of change of intensity values for pixels. A “high-frequency” may refer to a higher rate of change of the pixel intensity values, and a “low-frequency” may refer to a lower rate of change of the pixel intensity values. The terms “higher” and “lower” may be determined by referencing a threshold value, that may depend on the input raw image data, a type of sensor, a scene being captured, and so forth. Generally speaking, when the first resolution is 16K, the second resolution may be a resolution smaller than 16K (e.g., 14K, 10K, 8K, 4K, etc.). As another example, when the first resolution is 8K, the second resolution may be a resolution smaller than 8K (e.g., 4K, 2K, etc.).

[0059] Generally, a low-frequency portion of an image includes the color data of the image. The low-frequency portion may be obtained as a convolution of the image data with a Gaussian filter (e.g., a two-dimensional (2D) filter). Accordingly, high-frequency extraction involves subtracting the low-frequency convolution from the image data. Some embodiments involve storing, by the first portion of the ISP, the extracted high resolution information in a resolution extraction buffer. For example, HF extraction 335 A component may provide the extracted HF image data 340 to an HF buffer 345.

[0060] Some embodiments involve outputting, by the first portion of the ISP and for further image processing by the ISP, the modified input image having the second resolution. In some embodiments, resolution extraction circuitry (e.g., HF extraction 335 A) may be situated within ISP -BE 320. For example, HF extraction 335A may be situated between the Bayer processing circuitry and the temporal noise reduction circuitry. Also, for example, the modified input image having the second resolution (e.g., low-frequency portion) may be processed by ISP-BE 320. As illustrated, instead of processing high-frequency image data (as described with reference to FIG. 3), ISP-BE 320 can process low resolution (e.g., low-frequency) image data, as indicated by low-frequency image data (e.g., 4K) 330B. Note that in FIG. 3, the corresponding image data was high-resolution image data 330. Also, for example, components and intermediate buffers in ISP-BE 320 may now be configured to store low-frequency image data (e.g., 4K), resulting in lower consumption of memory resources. For example, low- frequency (LF) processor, such as YUV-LF, may now access and process low-frequency image data, resulting in lower consumption of memory resources and computational resources.

[0061] Also, for example, YUV-HF may access low-frequency image data, resulting in lower consumption of memory resources. As illustrated, ISP-BE 320 may be configured with a hardware resolution recovery (RR) 350 component. Some embodiments involve receiving aprocessed version of the modified input image after processing by the ISP, and retrieving the extracted high resolution information from the resolution extraction buffer. For example, RR 350 may retrieve the extracted high-frequency image data. Some embodiments involve performing resolution recovery by adding the extracted high resolution information to the processed version of the modified input image to generate an output image having the first resolution. For example, RR 350 may perform resolution recovery by combining the high- frequency image data 340 from HF buffer 345 with the low-frequency image data processed within ISP-BE 320. As described previously, RR 350 may output, for image post-processing, the output image having the first resolution. For example, a processed high-resolution image data may be output and saved in output buffer 355. One or more post-processing components of a camera system may access the processed high-resolution image data from output buffer 355.

[0062] As described with reference to FIG. 4, the image processing pipeline in FIG. 3 may be modified by adding the resolution extraction circuitry (e.g., HF extraction 335A) and a hardware RR 350 component as part of ISP-BE 320.

[0063] FIG. 5 illustrates another example of image processing with resolution recovery, in accordance with example embodiments. Although the position of hardware HF extraction 335A component in FIG. 4 is shown to be in ISP-BE 320, the HF extraction may be performed at other locations within ISP. For example, HF extraction may be performed as part of ISP-FE 310, subsequent to, or in parallel with, the processing in ISP-FE 310, and prior to processing within ISP-BE 320 (e.g., Bayer processing), and so forth. For example, the ISP may implement the high-frequency extraction block in the front-end of the ISP pipeline (e.g., ISP-FE 310) to save additional power. For example, this may result in DRAM BW saving (e.g., for intermediate buffer 315) and for ISP-BE 320. Also, for example, an area reduction may be achieved since the blocks after HF extraction may be designed up to a lower resolution (e.g., 4K), instead of a final higher resolution (e.g., 8K).

[0064] As illustrated in FIG. 5, ISP-FE 310 may provide raw image data in high-resolution (e.g., 8K) to HF extraction 335B. Although HF extraction 335B is illustrated to be located within ISP-FE 310, it may also be separate from ISP-FE 310. In some embodiments, HF extraction may be performed and the extracted HF image data 340 may be stored in HF buffer 345. Also, for example, in some embodiments, raw image data in low-resolution (e.g., 4K) may be provided back to ISF-FE 310 for further processing. In contrast to the arrangement illustrated in FIG. 4, intermediate buffer 315 in FIG. 5 receives initially processed raw imagedata in low-resolution (e.g., 4K) and this may be accessed by ISP-BE 320. For example, Bayer pattern processor, HDR processor, etc. may access the low-resolution image data. Subsequently, all processing of image data may be performed in low-resolution in a manner similar to that described with reference to FIG.4.

[0065] For example, ISP-BE 320 may be configured with a hardware resolution recovery (RR) 350 component. RR 350 may receive the extracted high-frequency image data 340 from HF buffer 345, and perform resolution recovery. For example, resolution recovery may be performed by RR 350 by combining the high-frequency image data 340 from HF buffer 345 with the low-frequency image data processed within ISP-BE 320.

[0066] Although FIG. 4 and FIG. 5 illustrate two example locations for HF extraction, additional and / or other locations for an HF extraction component are possible.

[0067] FIG. 6 illustrates high-frequency (HF) extraction 600, in accordance with example embodiments. Generally, the HF extraction component may be a high pass filter that outputs high-frequency image data. For example, HF extraction 600 can extract high-frequency information using downscaling (DS) block 615 and upscaling (US) block 620. In some embodiments, the high-frequency data may be extracted from an input image data 605 using a high-frequency extraction block (e.g., hardware HF extraction 335 A component of FIG. 4, hardware HF extraction 335B component of FIG. 5, etc.) by downscaling (DS) and upscaling (US) the input image data 605. In some embodiments, the input image data 605 may be luma. In the event the luma is linear, gamma conversion 610 may be applied to input image data 605. Intermediate buffer 315 is configured to maintain high data precision (e.g., 14K, 16K, etc.). Gamma conversion 610 produces an output that may be generally 8K.

[0068] Subsequently, DS block 615 and US block 620 may be applied to the image data after gamma conversion 610 is performed. The downscaling and upscaling are generally configured to align with the resolution recovery to be performed (e.g., by RR 350). Without such alignment, RR 350 may be unable to match the high-frequency image data 340 from HF buffer 345 with the processed low-frequency image data processed by ISP-BE 320.

[0069] In some embodiments, a difference 625 of the downscaled and upscaled image data may be determined with the original input image resolution (Y) that yields the high-frequency data. In some embodiments, a dynamic range compression (DRC) 630 may be applied to reduce the range of the high-frequency data. Subsequently, HF data may be written out and stored in HF buffer 345.

[0070] FIG. 7 illustrates resolution recovery 700, in accordance with example embodiments. For example, the final resolution may be generated using a resolution recovery block (e.g., RR 350 in ISP-BE 320) at the back-end of the ISP pipeline. As illustrated in FIG. 4 and FIG.5, RR 350 may receive extracted high-frequency image data 340 from HF buffer 345 and processed low-frequency image data processed by ISP-BE 320. In some embodiments, the low-frequency image data may be upscaled by US 730 to a target resolution by applying an appropriate scaling factor (e.g. x2 to convert 4K to 8K, or x4 to convert 4K to 16K, etc.). As described previously, the upscaling performed by US 730 of FIG. 7 as part of resolution recovery 700 has to match with the upscaling performed at US 620 of FIG. 6 as part of HF extraction to correctly align the low and high-frequency components. However, this upsampled data from US 730 is likely to lack high-frequency information.

[0071] In some embodiments, the high-frequency image data 340 may be optionally decompressed at block 710, adjusted by a tunable threshold at block 715 to adjust noise levels, and a gain may be applied at block 720. The thresholding at block 715 and the gain applied at block 720 may be adjusted to conform to desired image quality characteristics. The adjusted high-frequency data may be added to the upsampled image data from US 730 by adding component 735. Such a process can add the high-frequency information to the upsampled data from US 730. In some embodiments, the output of the adding component 735 may be clamped by clamp 740 to ensure that there is no overflow and the output conforms to the desired range. Subsequently, the higher resolution (e.g., 8K) data may be output to output buffer 355 for further image processing.

[0072] As described herein, both upscaling and downscaling within HF extraction, and the upscaling in resolution recovery have to match for frequency components to be aligned correctly. Also, for example, a choice of algorithm for upscaling and downscaling is maintained to be the same across these various blocks. For example, if cubic or bilinear interpolation is used for any one of these upscaling / downscaling processes, the same interpolation technique is used for all the upscaling / downscaling processes. Also, for example, upscaling / downscaling processes may be applied to chroma values, and / or to the entire image raw data. Such choices may be based on a trade-off between power savings and desired image quality.

[0073] FIG. 8 illustrates example images based on image processing with and without resolution recovery, in accordance with example embodiments. First image 805 illustrates an image with significantly reduced detail and resolution. Second image 810 illustrates the same image with significantly enhanced detail and resolution after a high-resolution extraction andresolution recovery process. For example, the reference numerals “12,” “14,” “16,” “18,” and “20” are sharper in second image 810 as compared to first image 805. As another example, second image 810 has more vertical lines and texture at reference numeral “18” (indicated by the region within the ellipse) than the corresponding portion in first image 805 (indicated by the region within the ellipse). Accordingly, even though processing is performed with a lower resolution image, the high resolution aspects are preserved in the HF extraction and resolution recovery process.

[0074] Although FIGs. 3-5 illustrate image processing for a Bayer pattern sensor, image processing may be performed based on various non-Bayer types of image sensors. A Bayer pattern based image sensor is generally used, and an ISP is also generally designed for Bayer patterns. However, use of higher resolution (e.g., 8K) image sensors has increased. Such sensors may utilize a quad Bayer pattern, where each 2x2 array of pixels have a same single color.

[0075] FIG. 9 illustrates an example of image processing 900 with a non-Bayer processing block, in accordance with example embodiments. FIG. 9 shares one or more aspects in common with FIGs. 3, 4, and / or 5. For example, input image may be received from an image sensor 305 for image processing. In some embodiments, input image may be received from a non-Bayer sensor. For example, image sensor 305 may be based on a quad Bayer pattern 305B or an RGBW pattern 305C. As illustrated, in the quad Bayer pattern 305B each 2x2 array of pixels have a same single color. Also, for example, in the RGBW pattern 305C, each 2x2 array of pixels, from left to right, starting at the top left corner of the illustrated grid, have a pattern of alternating 2x2 array of pixels of “WG”, followed by a 2x2 array of pixels of “WR”. At the next row, there is a pattern of 2x2 array of pixels of “WB”, followed by a 2x2 array of pixels of“WG”

[0076] One way to process the quad Bayer pattern 305B may be to interpolate the quad Bayer pattern 305B into a Bayer pattern by applying a remosaic process. After the remosaic process, the resulting raw image data may be processed as a Bayer pattern raw image data. The term “remosaic” as used herein generally refers to a rearrangement of the color filter arrays in a non- Bayer pattern into a Bayer pattern. However, applying such an approach results in data paths and buffers that are at a high-resolution (e.g., 8K), and such image processing generally results in a higher power consumption. Additional details of FIG. 9 are provided below after the description of FIGs. 10-12.

[0077] FIG. 10 illustrates an example color filter array (CFA) pattern and frequency spectrum for a Bayer pattern and a quad Bayer pattern, in accordance with example embodiments. For example, input image 1005 is associated with a CFA pattern and frequency spectrum. Bayer pattern image 1010 illustrates the CFA pattern and frequency spectrum for the image received by a Bayer sensor, and quad Bayer pattern image 1015 illustrates the CFA pattern and frequency spectrum for the image received by a quad Bayer sensor. As a result of the pattern characteristics of the image raw data in a quad Bayer pattern image 1015, there is generally little or no improvement in chroma frequency, and there may be increased aliasing artifacts, as indicated by the respective locations of the circles in the frequency spectrum for Bayer pattern image 1010 and quad Bayer pattern image 1015.

[0078] As described herein, an alternate manner of processing a non-Bayer image may be to perform binning on non-Bayer sensor patterns (e.g., quad Bayer pattern 305B, RGBW pattern 305C, etc.). Subsequently, the outcome of the binning may be sent to an image signal processor front-end (ISP-FE) pipeline. The term ’’binning” as used herein may generally refer to aggregating neighboring pixel values. For example, pixel values in 2x2 arrays of pixels may be aggregated by summing the pixel values, or by applying some averaging scheme (e.g., a weighted average). The averaging may be performed based on a desired output. For example, it may be preferable for some output images to have a higher composition of a particular color (e.g., G), and the averaging may apply a higher weight to pixel values for that color (e.g., G) than to pixel values for other colors.

[0079] FIG. 11 A illustrates an example binning applied to a quad Bayer pattern, in accordance with example embodiments. A first transformation 1105 may be applied to a 2x2 array of green pixels to convert the 2x2 array to a single green pixel. A second transformation 1110 may be applied to a 2x2 array of red pixels to convert the 2x2 array to a single red pixel. A third transformation 1115 may be applied to a 2x2 array of blue pixels to convert the 2x2 array to a single blue pixel. Binning 1120 may be based on applying the first transformation 1105, second transformation 1110, and third transformation 1115, to transform a quad Bayer pattern to a Bayer pattern.

[0080] FIG. 11B illustrates example chroma alias artifacts for a quad Bayer pattern after binning and after remosaicing, in accordance with example embodiments. For example, remosaic 1130 illustrates an image based on remosaicing. As indicated by the portion within the elliptical shape, several chroma alias artifacts may be observed. However, binning 1125 illustrates the same image based on binning and fewer chroma alias artifacts are observed.

[0081] FIG. 12 illustrates chroma noise artifacts for a quad Bayer pattern after binning and after remosaicing, in accordance with example embodiments. Generally, in remosaicing, a pixel’s effective pixel size is a quarter of the pixel size as compared with a binning mode (e.g., regular Bayer mode). This can result in a significant increase in chroma noise (e.g., in low light situations). After performing remosaicing at the beginning of the ISP pipeline, the effective kernel size of the noise reduction block is spatially reduced by A. For example, remosaic 1210 illustrates an image based on remosaicing with significantly more chroma noise artifacts as compared to binning 1205 that illustrates an image based on binning.

[0082] Accordingly, by applying an in-sensor remosaicing approach to non-Bayer patterns, there may be an increased power consumption due to the processing of high-frequency (e.g., 8K) data. Also, as illustrated, quality degradations may result, especially in chroma aliasing and chroma noise levels.

[0083] In some embodiments, the image sensor includes a non-Bayer pattern. The resolution extraction circuitry may include a non-Bayer processing circuitry. The high resolution information may include high-frequency information in a luma domain. The non-Bayer processing circuitry may be configured to extract the high-frequency information in the luma domain, while maintaining chroma values to match image data associated with a Bayer pattern.

[0084] Referring again to FIG. 9, a non-Bayer processing block 335C may be added. Some embodiments may involve a binning circuitry to apply binning to the input image. For example, non-Bayer processing block 335C may perform binning, and transmit the binned Bayer data to ISP -BE 320. Also, for example, luma high-frequency extraction 340B may be performed, and the image data may be output to luma buffer 345A (e.g., dynamic random access memory (DRAM) or single-level cell (SLC) memory). Also, for example, non-Bayer processing block 335C may provide lower resolution (e.g., 4K) data to intermediate buffer 315. This enables lower resolution data to be processed by ISP-BE 320. This saves the power consumptions typically associated with higher resolution (e.g., 8K) data processed in the remosaicing approach.

[0085] A luma resolution recovery 350B component may receive the information related to luma extraction 340B from luma buffer 345A. Luma resolution recovery may be performed by RR 350B in a manner similar to the process described with reference to FIG. 7. Subsequently, the higher resolution (e.g., 8K) data may be output to output buffer 355 for further image processing. The remaining operations of ISP-FE 310 and ISP-BE 320 may be similar to those described with reference to FIGs. 4-6 (e.g., for high-frequency extraction).

[0086] FIG. 13 illustrates an example of image processing 1300 with binning and luma high- frequency extraction, in accordance with example embodiments. For example, luma high- frequency extraction may be performed and the result may be sent to a memory (e.g., DRAM, SLC, etc.). Also, for example, high-frequency data may be computed in the luma domain by maintaining the same chroma as regular Bayer data. Accordingly, the ISP-FE pipeline may process the image in a manner similar to the regular Bayer data in the binned resolution, thereby resulting in significant reductions in the power and / or area. In some embodiments, a resolution recovery block may be used in an ISP backend pipeline (e.g., ISP-BE 320) to recover luma high-frequency.

[0087] For illustrative purposes, the non-Bayer process described herein is generally for a quad Bayer pattern. However, a similar approach may be applied to other non-Bayer patterns. Input image 1305 may be received by a non-Bayer processing component (e.g., non-Bayer processing block 335C of FIG. 9). Raw data for input image 1305 may be from a non-Bayer pattern sensor. For example, the non-Bayer pattern may be a quad Bayer pattern 1305 A or an RGBW pattern 1305B.

[0088] Although from an image filter point of view, the green pixels may be viewed as the same, the green pixels may differ from a pixel point of view. For example, some of the light on a green pixel may correspond to light that is diagonally incident and may have passed through neighboring filters of a different color (e.g., blue, red, etc.). This can introduce distortions in the image data among the green pixels. Accordingly, disparity correction 1310 may be applied to input image 1305.

[0089] Generally, quad Bayer data may involve additional processing such as shading correction 1320, and white balance gain 1325. First luma high-frequency extraction 1330 involves computing and extracting the luma high-frequency information, and high-frequency extraction 1345 involves high-frequency extraction as described herein.

[0090] Input image 1305 may also be provided to binning 1315 where binning may be performed. As described herein, binning reduces a resolution of the image data. In some embodiments, the binned image data may be provided to ISP-FE (e.g., ISP-FE 310).

[0091] Second luma high-frequency extraction 1335 may be performed on this low resolution image data, and chroma upsampling to a target image resolution may be performed by upsample 1340. In some aspects, for an RGBW pattern image data, luma high-frequency extraction (e.g., first luma high-frequency extraction 1330, second luma high-frequency extraction 1335) may involve extracting the color information from the color pixels whilemaintaining the information in the white pixels. Additional, and / or alternative extraction algorithms may be applied. High-frequency extraction 1345 may be performed on the high resolution data received from first luma high-frequency extraction 1330 and the upsampled image data received from second luma high-frequency extraction 1335. One or more aspects of high-frequency extraction 1345 may correspond to the processes described with reference to FIGs. 4-6. Subsequently, dynamic range compression (DRC) 1350 may be applied, and the output processed image may be provided to a chroma buffer 1355.Example Methods of Operation

[0092] FIG. 14 is a flowchart of a method 1400, in accordance with example embodiments. Method 1400 can be executed by a computing device, such as computing device 100.

[0093] At block 1410, the method involves obtaining, by a first portion of an image signal processor (ISP), an input image from an image sensor, the input image having a first resolution. In some examples, the operations of block 1410 may be performed by an image reception circuitry.

[0094] At block 1420, the method involves extracting, by the first portion of the ISP, high resolution information from the input image having the first resolution to generate a modified input image having a second resolution that is smaller than the first resolution.

[0095] At block 1430, the method involves outputting, by the first portion of the ISP and for further image processing by the ISP, the modified input image having the second resolution.

[0096] At block 1440, the method involves storing, by the first portion of the ISP, the extracted high resolution information in a resolution extraction buffer.

[0097] In some examples, the operations of blocks 1420-1440 may be performed by a resolution extraction circuitry situated at the first portion of the image signal processor (ISP).

[0098] At block 1450, the method involves receiving, by a second portion of the ISP, a processed version of the modified input image after processing by the ISP.

[0099] At block 1460, the method involves retrieving, by the second portion of the ISP, the extracted high resolution information from the resolution extraction buffer.

[0100] At block 1470, the method involves performing, by the second portion of the ISP, resolution recovery by adding the extracted high resolution information to the processed version of the modified input image to generate an output image having the first resolution.

[0101] At block 1480, the method involves providing, by the second portion of the ISP and for image post-processing, the output image having the first resolution.

[0102] In some examples, the operations of blocks 1460-1480 may be performed by a resolution recovery circuitry situated at the second portion of the ISP.

[0103] In some embodiments, the first portion may be situated within a front-end of the ISP, and a memory circuitry may be configured as an intermediate buffer between the front-end of the ISP and a back-end of the ISP, and the intermediate buffer may be configured to store the modified input image having the second resolution.

[0104] In some embodiments, the first portion may be situated within a back-end of the ISP.

[0105] Some embodiments involve a Bayer processing circuitry situated within a back-end of the ISP, and a temporal noise reduction circuitry within the back-end of the ISP. The resolution extraction circuitry may be situated between the Bayer processing circuitry and the temporal noise reduction circuitry.

[0106] In some embodiments, the resolution extraction circuitry may include a high-frequency extraction circuitry. The high resolution information may include high-frequency portions of the input image. The high-frequency extraction circuitry may be configured to extract the high- frequency portions from the input image. The modified input image may include low-frequency portions of the input image after the extraction of the high-frequency portions from the input image.

[0107] In some embodiments, the high-frequency extraction circuitry may be configured to downscale the input image to the second resolution, and upscale the downscaled input image to the first resolution. The high-frequency portions may be extracted based on a difference between the upscaled image and the input image.

[0108] In some embodiments, the input image may include a linear luma input. Such embodiments may include a gamma conversion circuitry configured to adjust pixel values of one or more pixels in the input image. The high-frequency portions may be extracted based on a difference between the upscaled image, and the input image after gamma conversion.

[0109] Some embodiments involve applying a dynamic range compression to reduce a range of the high-frequency portions.[HO] In some embodiments, the image sensor includes a non-Bayer pattern. The resolution extraction circuitry may include a non-Bayer processing circuitry. The high resolution information may include high-frequency information in a luma domain. The non-Bayer processing circuitry may be configured to extract the high-frequency information in the luma domain, while maintaining chroma values to match image data associated with a Bayer pattern. [Hl] Some embodiments involve a binning circuitry to apply binning to the input image.

[0112] In some embodiments, the binned input image may be provided to a front-end of the ISP.

[0113] Some embodiments involve a Bayer processing circuitry situated at a back-end of the ISP. The Bayer processing circuitry may be configured to process binned image data output by the front-end of the ISP.

[0114] In some embodiments, the non-Bayer pattern may be a quad Bayer pattern or an RGBW pattern.

[0115] Some embodiments involve upscaling the processed version of the modified input image having the second resolution to the first resolution. The performing of the resolution recovery involves adding the extracted resolution information from the resolution extraction buffer to the processed version of the modified image as upscaled.

[0116] Some embodiments involve applying a tunable thresholding and a gain to the extracted high resolution information. The adding of the extracted high resolution information comprises adding the extracted high resolution information after the applying of the tunable thresholding and the gain.

[0117] In some embodiments, the resolution recovery circuitry may be configured to decompress the extracted high resolution information.

[0118] In some embodiments, upscaling and downscaling algorithms may be the same.

[0119] In some embodiments, upscaling and downscaling algorithms may include one of a cubic interpolation or a bilinear interpolation.

[0120] The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its spirit and scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those enumerated herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.

[0121] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. It will bereadily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

[0122] With respect to any or all of the ladder diagrams, scenarios, and flow charts in the figures and as discussed herein, each block and / or communication may represent a processing of information and / or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, functions described as blocks, transmissions, communications, requests, responses, and / or messages may be executed out of order from that shown or discussed, including substantially concurrent or in reverse order, depending on the functionality involved. Further, more or fewer blocks and / or functions may be used with any of the ladder diagrams, scenarios, and flow charts discussed herein, and these ladder diagrams, scenarios, and flow charts may be combined with one another, in part or in whole.

[0123] A block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and / or related data may be stored on any type of computer readable medium such as a storage device including a disk or hard drive or other storage medium.

[0124] The computer readable medium may also include non-transitory computer readable media such as non-transitory computer-readable media that stores data for short periods of time like register memory, processor cache, and random access memory (RAM). The computer readable media may also include non-transitory computer readable media that stores program code and / or data for longer periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or nonvolatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

[0125] Moreover, a block that represents one or more information transmissions may correspond to information transmissions between software and / or hardware modules in thesame physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.

[0126] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for explanatory purposes and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMSWhat is claimed is:

1. A system comprising: an image reception circuitry configured to obtain an input image from an image sensor, the input image having a first resolution; a resolution extraction circuitry situated at a first portion of an image signal processor (ISP), the resolution extraction circuitry configured to: extract high resolution information from the input image having the first resolution to generate a modified input image having a second resolution that is smaller than the first resolution, output, for further image processing by the ISP, the modified input image having the second resolution, and store the extracted high resolution information in a resolution extraction buffer; and a resolution recovery circuitry situated at a second portion of the ISP, the resolution recovery circuitry configured to: receive a processed version of the modified input image after processing by the ISP, retrieve the extracted high resolution information from the resolution extraction buffer, perform resolution recovery by adding the extracted high resolution information to the processed version of the modified input image to generate an output image having the first resolution, and provide, for image post-processing, the output image having the first resolution.

2. The system of claim 1, wherein the first portion is situated within a front-end of the ISP, the system further comprising: memory circuitry configured as an intermediate buffer between the front-end of the ISP and a back-end of the ISP, the intermediate buffer configured to store the modified input image having the second resolution.

3. The system of any one of claims 1 or 2, wherein the first portion is situated within a back-end of the ISP.

4. The system of any one of claims 1-3, further comprising: a Bayer processing circuitry situated within a back-end of the ISP; and a temporal noise reduction circuitry within the back-end of the ISP, and wherein the resolution extraction circuitry is situated between the Bayer processing circuitry and the temporal noise reduction circuitry.

5. The system of any one of claims 1-4, wherein the resolution extraction circuitry comprises a high-frequency extraction circuitry, wherein the high resolution information comprises high-frequency portions of the input image, wherein the high-frequency extraction circuitry is configured to extract the high-frequency portions from the input image, and wherein the modified input image comprises low-frequency portions of the input image after the extraction of the high-frequency portions from the input image.

6. The system of claim 5, wherein the high-frequency extraction circuitry is configured to: downscale the input image to the second resolution; and upscale the downscaled input image to the first resolution, and wherein the high-frequency portions are extracted based on a difference between the upscaled image and the input image.

7. The system of claim 6, wherein the input image comprises a linear luma input, the system further comprising: a gamma conversion circuitry configured to adjust pixel values of one or more pixels in the input image, and wherein the high-frequency portions are extracted based on a difference between the upscaled image, and the input image after gamma conversion.

8. The system of claim 5, wherein the high-frequency extraction circuitry is configured to: apply a dynamic range compression to reduce a range of the high-frequency portions.

9. The system of any one of claims 1-8, wherein the image sensor comprises a non-Bayer pattern, wherein the resolution extraction circuitry comprises a non-Bayer processing circuitry, wherein the high resolution information comprises high-frequency information in a luma domain, and wherein the non-Bayer processing circuitry is configured to extract the high-frequency information in the luma domain, while maintaining chroma values to match image data associated with a Bayer pattern.

10. The system of claim 9, further comprising: a binning circuitry to apply binning to the input image.

11. The system of claim 10, wherein the binned input image is provided to a frontend of the ISP.

12. The system of claim 11, further comprising: a Bayer processing circuitry situated at a back-end of the ISP, the Bayer processing circuitry configured to process the binned image data output by the front-end of the ISP.

13. The system of claim 10, wherein the resolution extraction circuitry further comprises a high-frequency extraction circuitry, wherein the high resolution information further comprises high-frequency portions of the input image, and wherein the high-frequency extraction circuitry is further configured to extract the high-frequency portions from the input image.

14. The system of any one of claims 9-13, wherein the non-Bayer pattern is a quad Bayer pattern or an RGBW pattern.

15. The system of any one of claims 1-14, wherein the resolution recovery circuitry is configured to: upscale the processed version of the modified input image having the second resolution to the first resolution, andwherein the performing of the resolution recovery comprises adding the extracted resolution information from the resolution extraction buffer to the processed version of the modified image as upscaled.

16. The system of claim 15, wherein the resolution recovery circuitry is configured to: apply a tunable thresholding and a gain to the extracted high resolution information, and wherein the adding of the extracted high resolution information comprises adding the extracted high resolution information after the applying of the tunable thresholding and the gain.

17. The system of claim 15, wherein the resolution recovery circuitry is configured to: decompress the extracted high resolution information.

18. The system of any one of claims 1-17, wherein upscaling and downscaling algorithms are the same.

19. The system of any one of claims 1-18, wherein upscaling and downscaling algorithms comprise one of a cubic interpolation or a bilinear interpolation.

20. A computing device, comprising: a processor; and a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations of any of the circuitries in accordance with any of claims 1-19.

21. A computer-implemented method comprising: obtaining, by a first portion of an image signal processor (ISP), an input image from an image sensor, the input image having a first resolution;extracting, by the first portion of the ISP, high resolution information from the input image having the first resolution to generate a modified input image having a second resolution that is smaller than the first resolution; outputting, by the first portion of the ISP and for further image processing by the ISP, the modified input image having the second resolution; storing, by the first portion of the ISP, the extracted high resolution information in a resolution extraction buffer; receiving, by a second portion of the ISP, a processed version of the modified input image after processing by the ISP; retrieving, by the second portion of the ISP, the extracted high resolution information from the resolution extraction buffer; performing, by the second portion of the ISP, resolution recovery by adding the extracted high resolution information to the processed version of the modified input image to generate an output image having the first resolution; and providing, by the second portion of the ISP and for image post-processing, the output image having the first resolution.

Images