Directional scaling system and method

Abstract

The invention relates to a directional scaling system and method. An electronic device may include scaling circuitry to scale input pixel data to a greater resolution. The directional scaling circuit may include a first interpolation circuit to receive optimal mode data, the optimal mode data including one or more angles corresponding to content of an image, and interpolating a first pixel value at a first pixel position diagonally offset from an input pixel position from the input pixel data based on the optimal mode data and an input pixel value corresponding to the input pixel position. The directional scaling circuit may also include a second interpolation circuit to receive the optimal mode data and the input pixel value, and to interpolate a second pixel value based at least in part on the optimal mode data and the input pixel value at a second pixel position that is horizontally or vertically offset from the input pixel position.

Description

[0001] This application is a divisional application of the invention patent application with the application date of July 24, 2019, the application number of 202210384692.9, and the name of “Directional Zooming System and Method”. BACKGROUND

[0002] The present disclosure relates generally to image processing, and more particularly to the analysis of pixel statistics, zooming, and / or enhancement of image data for displaying images on electronic displays.

[0003] This section is intended to introduce the reader to various aspects of art that can be related to various aspects of the present disclosure and is not intended to limit the scope of the present disclosure. The discussion below is intended to provide context for the technology described herein and to assist the reader in understanding the aspects of the technology disclosed below. Accordingly, the statements made herein are not to be construed as admissions about what is or is not considered to be prior art.

[0004] Electronic devices often present visual representations of information, such as text, still images, and / or video, by displaying one or more images (e.g., image frames) using one or more electronic displays. For example, such electronic devices can include computers, mobile telephones, portable media devices, tablet computers, televisions, virtual reality headsets, and vehicle dashboards, among others. To display an image, an electronic display can control the light emission (e.g., luminance) of its display pixels based at least in part on corresponding image / pixel data.

[0005] Generally, image data can indicate a resolution corresponding to an image (e.g., a size of pixels to be used). However, in some cases, it can be desirable to zoom an image to a higher resolution, such as for display on an electronic display having a higher resolution output. Accordingly, image data can be processed to convert the image data to a desired resolution before it is used to display an image. However, at least in some cases, techniques for zooming image data can affect the perceived image quality of a corresponding image, such as by introducing image artifacts like jagged edges. Pixel statistics can be employed to correct for such artifacts when image enhancement is experienced. SUMMARY

[0006] A summary of certain implementations disclosed herein is set forth below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of these certain implementations and that the aspects are not intended to limit the scope of the present disclosure. Indeed, the present disclosure can encompass a variety of aspects that can not be set forth below.

[0007] In some cases, an electronic device can scale and / or enhance image data to improve the perceived quality of an image. In some embodiments, the alterations to the image data can be based at least in part on the content of the image corresponding to the image data. Such altered image data can be stored in memory or displayed on an electronic display. In some embodiments, the image data can indicate a target luminance per color component (e.g., channel) via red, blue, and green component image data, for example. Additionally or alternatively, the image data can indicate a target luminance in grayscale (e.g., gray scale) or via luminance and chrominance components (e.g., YCbCr).

[0008] To facilitate improvements to the image data, noise statistics can be collected and analyzed. For example, frequency bands within the image data can be identified to assist in distinguishing image content from noise. As such, the content of the image can be enhanced while minimizing the impact of noise on the output image. In some embodiments, pixels that do not satisfy certain criteria can be excluded from the noise statistics.

[0009] Additionally, difference statistics and sum of absolute differences (SAD) can be applied to pixel groupings of the image data. Such pixel groupings can be selected and compared in multiple directions relative to a pixel of interest via an angle detection circuit to identify a best mode (e.g., angle) for interpolation. Comparisons of different pixel groupings can identify features (edges, lines, and / or changes) in the image content that can assist in using the best mode to enhance or scale the image data. The best mode data can include one or more angles that most accurately describe the features of the image content at the location of the pixel of interest. Additionally, the best mode data from multiple pixels of interest can be compiled together for pixel value interpolation.

[0010] For example, in one embodiment, the difference statistics and SAD statistics can facilitate an increase in image resolution by interpolating new pixel values based at least in part on the best mode data. A directional scaling circuit can utilize the identified angles to maintain features of the image while minimizing the introduction of artifacts such as jagged edges (e.g., jaggies). In some embodiments, the directional scaling circuit can interpolate pixels that are diagonal to the original image data pixel locations by generating a weighted average of the original pixels. The weighted average can be based on the difference statistics and SAD statistics and the angles identified therefrom, for example. Additionally, the directional scaling circuit can generate horizontally and vertically positioned pixel values from the original pixels by determining a weighted average of the original pixels and / or the new diagonal pixels. The weighted average can also be based at least in part on the difference statistics and SAD statistics.

[0011] The enhancement circuit, in combination with the directional scaling circuit or separately, can also use the difference statistics and SAD statistics to adjust the image data of the image. Additionally or alternatively, the enhancement circuit can use noise statistics values and / or a low resolution version of the image to generate image enhancements. Such enhancements can provide increased sharpness to the image. In some embodiments, example-based enhancements using a lower resolution version of the image for comparison can provide enhancements to one or more channels of the image data. For example, a luminance channel of the image data can be enhanced based on the sum of squared differences or an approximation thereof between the image and a low resolution version of the image. Additionally, the enhancement circuit can employ a peak filter to enhance high frequency aspects of the image (e.g., crosshatching). Such enhancements can provide improved spatial resolution and / or reduced blurring. Additionally, the enhancement circuit can determine the hue within the image to identify certain content (e.g., sky, grass, and / or skin). The example-based enhancements, peak filter, and / or hue determination can each target different textures of the image to combine enhancements, and the enhancements resulting from each can be controlled independently and based on local features of the image.

[0012] Depending on the implementation, the noise statistics circuit, the angle detection circuit, the directional scaling circuit, and the enhancement circuit can be used individually and / or in combination to facilitate improved perceived image quality and / or to alter the image data to a higher resolution while reducing the likelihood of image artifacts. BRIEF DESCRIPTION OF DRAWINGS

[0013] Various aspects of the disclosure can be better understood when read in conjunction with the following detailed description and with reference to the drawings, in which:

[0014] Figure 1 is a block diagram of an electronic device including an electronic display in accordance with an embodiment;

[0015] Figure 2 is an example of the electronic device of Figure 1 in accordance with an embodiment;

[0016] Figure 3 is another example of the electronic device of Figure 1 in accordance with an embodiment;

[0017] Figure 4 is another example of the electronic device of Figure 1 in accordance with an embodiment;

[0018] Figure 5 is another example of the electronic device of Figure 1 in accordance with an embodiment;

[0019] Figure 6 is a block diagram of a processing pipeline of a memory of the electronic device of Figure 1 in accordance with an embodiment;

[0020] Figure 7 It is based on the implementation plan and can be made by Figure 1 A block diagram of the scaler block used by electronic devices;

[0021] Figure 8 It is for operation according to the implementation plan. Figure 7 A flowchart of the scaling block process;

[0022] Figure 9 It is based on the implementation plan. Figure 7 A block diagram of the noise statistics block implemented in the scaler block;

[0023] Figure 10 It is for operation according to the implementation plan. Figure 9 The flowchart of the noise statistics block process;

[0024] Figure 11 It is based on the implementation plan. Figure 7 A block diagram of the angle detection block implemented in the scaler block;

[0025] Figure 12 This is a schematic diagram of the pixel positions and exemplary sampling according to the implementation scheme;

[0026] Figure 13 This is a schematic diagram of the pixel positions and exemplary sampling according to the implementation scheme;

[0027] Figure 14 It is for operation according to the implementation plan. Figure 11 A flowchart of the angle detection block process;

[0028] Figure 15 It is based on the implementation plan. Figure 7 A block diagram of the directional scaler block implemented in the scaler block;

[0029] Figure 16 This is a schematic diagram of exemplary pixel interpolation points according to the implementation scheme;

[0030] Figure 17 It is for operation according to the implementation plan. Figure 15 A flowchart of the process of the orientation calibrator block;

[0031] Figure 18 Is Figure 7 A block diagram of the image enhancement block implemented in the scaler block;

[0032] Figure 19 Is Figure 18 An improved block diagram based on examples implemented in the image enhancement block; and

[0033] Figure 20 It is for operation according to the implementation plan.Figure 18 The flowchart of the enhancement block process. Detailed Implementation

[0034] One or more specific implementations will be described below. To provide a brief description of these implementations, not all characteristics of the actual implementations are described in this specification. It should be understood that in the development of any such actual implementation, as in any engineering or design project, decisions must be made specific to many implementations to achieve the developer's specific objectives, such as compliance with system-related and business-related constraints that may vary from one implementation to another. Furthermore, it should be understood that such development work can be complex and time-consuming, but will still be routine work of design, fabrication, and manufacturing for those skilled in the art who benefit from this disclosure.

[0035] To facilitate the communication of information, electronic devices typically use one or more electronic displays to present a visual representation of the information via one or more images (e.g., image frames). Such electronic devices may include computers, mobile phones, portable media devices, tablets, televisions, virtual reality headsets, and vehicle dashboards, among others. Additionally or alternatively, electronic displays may take the form of liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic light-emitting diode (OLED) displays, plasma displays, and the like.

[0036] In any case, to display an image, an electronic display typically controls the light emission (e.g., brightness and / or color) of its display pixels based on corresponding image data received at a specific resolution (e.g., pixel size). For example, an image data source (e.g., memory, input / output (I / O) port, and / or communication network) may output image data as a stream of pixel data (e.g., image data), where the data for each pixel indicates the target brightness (e.g., luminance and / or color) of one or more display pixels located at the corresponding pixel location. In some embodiments, the image data may indicate the brightness of each color component, for example, via red component image data, blue component image data, and green component image data (collectively, RGB). Additionally or alternatively, the image data may be indicated by a luminance channel and one or more chrominance channels (e.g., YCbCr, YUV, etc.), grayscale (e.g., gray levels), or other color bases. It should be understood that, as disclosed herein, the luminance channel may encompass linear luminance values, non-linear luminance values, and / or gamma-corrected luminance values.

[0037] To facilitate improved perceived image quality, image data can be processed prior to being output to an electronic display or stored in memory for later use. For example, a processing pipeline implemented via hardware (e.g., circuitry) and / or software (e.g., execution of instructions stored in a tangible, non-transitory medium) can facilitate such image processing. In some cases, it can be desirable to scale image data to a higher resolution, for example to match the resolution of an electronic display or to make an image appear larger. However, at least in some cases, this can impact perceived image quality, for example by causing perceptible visual artifacts such as blurring, jagged edges (e.g., jaggies), and / or loss of detail.

[0038] Accordingly, to facilitate improved perceived image quality, the present disclosure provides techniques for identifying content of an image (e.g., via statistical values), scaling image data to increase resolution while maintaining image sharpness (e.g., acuity), and / or enhancing image data to increase sharpness of an image. In some embodiments, a processing pipeline can include a scaler block to directionally scale image data while taking into account lines, edges, patterns, and angles within the image. Such content-dependent processing can allow image data to be scaled to a higher resolution without artifacts or with a reduced amount of artifacts. In one embodiment, the ability to increase image resolution without introducing noticeable artifacts can allow an image to be stored at a lower resolution, saving memory space and / or bandwidth, and restored to a higher resolution prior to displaying the image. Furthermore, the image data can undergo further enhancement prior to being output or stored.

[0039] To seek such content-dependent processing, the scaler block can include, for example, a noise statistics block, an angle detection block, a directional scaling block, and an image enhancement block. The noise statistics block can use statistical analysis of collected pixel statistics values to identify noise and distinguish noise from the rest of the image data. As such, if / when the image data undergoes enhancement, the noise can be ignored or given less weight. The angle detection block can collect statistics values based on sum of absolute differences (SAD) and / or differences (DIFF). These SAD statistics values and DIFF statistics values can be determined at multiple angles around an input pixel to identify an angle of interest from which to perform a basic scaling interpolation and / or enhancement. Thus, the best mode data identified for each input pixel (including, for example, a best angle, a weight, etc.) can assist in characterizing the image content to help directional scaling of the image data. The directional scaling block can take the input image data and the best mode data and interpolate midpoint pixels and outer point pixels to generate new pixel data to add to the input image data to generate scaled image data. The scaled image data can further be enhanced via the image enhancement block by identifying tones within the image and by comparing the scaled image data to the input image data via example-based improvement. As such, the scaler block can incorporate hardware components and / or software components to facilitate determining noise and angles of interest, scaling image data to higher resolutions while reducing the likelihood of image artifacts, and / or image enhancement.

[0040] To help illustrate, Figure 1 An electronic device 10 is shown that can include an electronic display 12. As will be described in greater detail below, the electronic device 10 can be any suitable electronic device 10, such as a computer, a mobile phone, a portable media device, a tablet computer, a television, a virtual reality headset, a vehicle dashboard, etc. Thus, it should be noted that, Figure 1 is merely one example of a particular implementation and is intended to illustrate the types of components that can be present in an electronic device 10.

[0041] In the depicted implementation, the electronic device 10 includes an electronic display 12, one or more input devices 14, one or more input / output (I / O) ports 16, a processor core complex 18 having one or more processors or processor cores, a local memory 20, a main memory storage device 22, a network interface 24, a power supply 26, and image processing circuitry 27. Figure 1The various components described in the middle can include hardware elements (e.g., circuitry), software elements (e.g., tangible, non-transitory computer-readable media storing instructions), or a combination of both hardware and software elements. It should be noted that the various depicted components can be combined into fewer components or separated into additional components. For example, local memory 20 and main memory storage device 22 can be included in a single component. Additionally or alternatively, image processing circuitry 27 (e.g., a graphics processing unit) can be included in processor core complex 18.

[0042] As shown, processor core complex 18 is operatively coupled with local memory 20 and main memory storage device 22. Accordingly, processor core complex 18 can execute instructions stored in local memory 20 and / or main memory storage device 22 to perform operations such as generating and / or transmitting image data. As such, processor core complex 18 can include one or more general-purpose microprocessors, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), or any combination thereof.

[0043] In addition to instructions, local memory 20 and / or main memory storage device 22 can store data to be processed by processor core complex 18. Accordingly, in some embodiments, local memory 20 and / or main memory storage device 22 can include one or more tangible, non-transitory computer-readable media. For example, local memory 20 can include random access memory (RAM), and main memory storage device 22 can include read only memory (ROM), rewritable non-volatile memory (such as flash memory, hard drives, optical discs, and / or the like).

[0044] As shown, processor core complex 18 is also operatively coupled with network interface 24. In some embodiments, network interface 24 can facilitate data communication with another electronic device and / or a communication network. For example, network interface 24 (e.g., a radio frequency system) can enable electronic device 10 to be communicatively coupled to a personal area network (PAN), such as a Bluetooth network, a local area network (LAN) (such as an 802.1 lx Wi-Fi network), and / or a wide area network (WAN) (such as a 4G or LTE cellular network).

[0045] Further, as shown, processor core complex 18 is operatively coupled to power supply 26. In some embodiments, power supply 26 can provide power to one or more components in electronic device 10, such as processor core complex 18 and / or electronic display 12. Accordingly, power supply 26 can include any suitable energy source, such as a rechargeable lithium polymer (Li-poly) battery and / or an alternating current (AC) power converter.

[0046] Additionally, as shown, the processor core complex 18 is operatively coupled with one or more I / O ports 16. In some embodiments, the I / O ports 16 can enable the electronic device 10 to interface with other electronic devices. For example, when a portable storage device is connected, the I / O ports 16 can enable the processor core complex 18 to communicate data with the portable storage device.

[0047] As shown, the electronic device 10 is also operatively coupled with one or more input devices 14. In some embodiments, the input devices 14 can facilitate user interaction with the electronic device 10 by, for example, receiving user input. Thus, the input devices 14 can include buttons, keyboards, mice, touchpads, etc. Additionally, in some embodiments, the input devices 14 can include a touch-sensing component in the electronic display 12. In such embodiments, the touch-sensing component can receive user input by detecting the occurrence and / or location of an object touching the surface of the electronic display 12.

[0048] In addition to enabling user input, the electronic display 12 can include a display panel having one or more display pixels. The electronic display 12 can control light emission from its display pixels to thereby present a visual representation of information, such as a graphical user interface (GUI) of an operating system, an application interface, a still image, or video content, by displaying a frame based at least in part on corresponding image data (e.g., image pixel data at each pixel location).

[0049] As shown, the electronic display 12 is operatively coupled to the processor core complex 18 and the image processing circuit 27. In this manner, the electronic display 12 can display an image based at least in part on image data received from an image data source, such as the processor core complex 18 and / or the image processing circuit 27. In some embodiments, the image data source can generate source image data to create a digital representation of an image to be displayed. In other words, the image data is generated such that the view on the electronic display 12 accurately represents the intended image. Additionally or alternatively, the electronic display 12 can display an image based at least in part on image data received via the network interface 24, the input devices 14, and / or the I / O ports 16. To facilitate accurate representation of an image, the image data can be processed, e.g., via a processing pipeline and / or a display pipeline implemented in the processor core complex 18 and / or the image processing circuit 27, prior to being supplied to the electronic display 12. Further, in some embodiments, the image data can be obtained, e.g., from the memory 20, processed, e.g., in a processing pipeline, and returned to its source, e.g., the memory 20. As described herein, such techniques are referred to as memory-to-memory processing.

[0050] As will be described in greater detail below, the processing pipeline can perform various processing operations, such as image scaling, rotation, enhancement, pixel statistics gathering and interpretation, spatial and / or temporal dithering, pixel color space conversion, brightness determination, brightness optimization, etc. For example, when displaying a corresponding image on the electronic display 12, the processing pipeline can directionally scale the image data to increase resolution while using the content of the image data to reduce the likelihood of producing perceptible visual artifacts (e.g., jagged edges, banding, and / or blurring).

[0051] In some embodiments, after receiving the image data, the electronic display 12 can perform additional processing on the image data, e.g., to further improve the accuracy of the viewed image. For example, the electronic display 12 can again scale, rotate, spatially dither, and / or enhance the image data. As such, in some embodiments, the processing pipeline can be implemented with the electronic display 12.

[0052] As noted above, the electronic device 10 can be any suitable electronic device. For ease of illustration, one example of a suitable electronic device 10, particularly a handheld device 10A, is shown in FIG. 1. In some embodiments, the handheld device 10A can be a portable telephone, a media player, a personal data organizer, a handheld gaming platform, etc. For illustrative purposes, the handheld device 10A can be a smartphone, such as any iPhone® model available from Apple Inc. Figure 2

[0053] As shown, the handheld device 10A includes a housing 28 (e.g., a case). In some embodiments, the housing 28 can protect the internal components from physical damage and / or shield the internal components from electromagnetic interference. Additionally, as shown, the housing 28 can enclose the electronic display 12. In the depicted embodiment, the electronic display 12 displays a graphical user interface (GUI) 30 having an array of icons 32. For example, when an icon 32 is selected by the input device 14 or a touch-sensing component of the electronic display 12, an application can be launched.

[0054] Furthermore, as shown, the input device 14 can be accessible through an opening in the housing 28. As noted above, the input device 14 can enable a user to interact with the handheld device 10A. For example, the input device 14 can enable a user to activate or deactivate the handheld device 10A, navigate the user interface to a home screen, navigate the user interface to a user-configurable application screen, activate a voice recognition feature, provide volume control, and / or toggle between vibrate and ringer modes. As shown, the I / O port 16 can be accessible through an opening in the housing 28. In some embodiments, the I / O port 16 can include, for example, an audio jack to connect to an external device. ​​

[0055] To further illustrate, another example of a suitable electronic device 10, particularly a tablet device 10B, is shown below. Figure 3 For illustrative purposes, the tablet device 10B may be any device available from Apple Inc. Model number. Figure 4 Another example of a suitable electronic device 10 is shown, specifically a computer 10C. For illustrative purposes, the computer 10C may be any device available from Apple Inc. or Model. Another example of a suitable electronic device 10, especially a watch 10D, is shown below. Figure 5 For illustrative purposes, the Watch 10D is available from any Apple Inc. Model. As shown in the figure, the tablet device 10B, computer 10C and watch 10D each also include an electronic display 12, an input device 14, an I / O port 16 and a housing 28.

[0056] As described above, the electronic display 12 can display an image based on image data received from an image data source. For ease of explanation, Figure 6 A portion 34 of an electronic device 10 is shown, comprising a processing pipeline 36 that operatively retrieves, processes, and outputs image data. In some embodiments, the processing pipeline 36 may analyze and / or process image data received from memory 20, for example, by directionally scaling and enhancing the image data before it is used to display an image or stored in memory 20, as in memory-to-memory processing. In this context, the image data may be directionally scaled to a higher resolution and then stored in memory for later viewing. In some embodiments, the processing pipeline 36 may be incorporated into or integrated into a display pipeline and is therefore operatively coupled to a display driver 38 to generate analog and / or digital electrical signals, at least in part, based on the image data and supply them to the display pixels of the electronic display 12.

[0057] In some implementations, the processing pipeline 36 may be implemented in the electronic device 10, the electronic display 12, or a combination thereof. For example, the processing pipeline 36 may be included in the processor core complex 18, the image processing circuitry 27, the timing controller (TCON) in the electronic display 12, one or more other processing units or circuits, or any combination thereof.

[0058] In some embodiments, controller 40 can control operation of processing pipeline 36, memory 20, and / or display driver 38. To facilitate control operations, controller 40 can include a controller processor and a controller memory. In some embodiments, the controller processor can execute instructions stored in the controller memory, such as firmware. In some embodiments, the controller processor can be included in processor core complex 18, image processing circuit 27, a timing controller in electronic display 12, a separate processing module, or any combination thereof. Additionally, in some embodiments, the controller memory can be included in local memory 20, main memory storage device 22, a separate tangible, non-transitory computer-readable medium, or any combination thereof.

[0059] In some embodiments, memory 20 can include a source buffer that stores source image data. Accordingly, in such embodiments, processing pipeline 36 can fetch (e.g., retrieve) source image data from the source buffer for processing. In some embodiments, electronic device 10 can include multiple pipelines (e.g., processing pipeline 36, display pipeline, etc.) implemented to process image data. To facilitate communication, image data can be stored in memory 20 external to the pipelines. In such embodiments, a pipeline (e.g., processing pipeline 36) can include a direct memory access (DMA) block that reads (e.g., retrieves) and / or writes (e.g., stores) image data in memory 20.

[0060] Upon receipt from memory 20, processing pipeline 36 can process source image data via one or more image processing blocks, such as scaling and rotation block 42 or other processing block 44 (e.g., dithering block). In the depicted embodiment, scaling and rotation block 42 includes a scaler block 46 and can also include other modification blocks 48 (e.g., rotation blocks, flip blocks, mirror blocks, etc.). As will be described in greater detail below, scaler block 46 can adjust image data (e.g., via directional scaling and / or enhancement), for example, to facilitate reducing the likelihood of or correcting for image artifacts typically associated with scaling. As an illustrative example, it can be desirable to increase the resolution of image data to expand the viewing of a corresponding image or to accommodate the resolution of electronic display 12. To accomplish this, scaler block 46 can employ noise statistics and / or SAD and DIFF statistics to analyze the content of image data and scale the image data to a higher resolution while maintaining image sharpness (e.g., acuity). In some embodiments, image data can also undergo enhancement.

[0061] To facilitate illustration, Figure 7is a block diagram of a scaler block 46 that receives input image data 50 and outputs processed image data 52. The scaler block 46 can include a plurality of processing blocks 54 to perform directional scaling and / or enhancement. For example, the scaler block 46 can include a transform block 56, a noise statistics block 58, an angle detection block 60, a directional scaling block 62, an image enhancement block 64, and a vertical / horizontal scaling block 66.

[0062] The processing blocks 54 of the scaler block 46 can receive and / or process the input image data 50 in a plurality of color bases (e.g., red-green-blue (RGB), alpha-red-green-blue (ARGB), luminance-chrominance (YCC formats such as YCbCr or YUV), etc.) and / or bit depths (e.g., 8-bit, 16-bit, 24-bit, 30-bit, 32-bit, 64-bit, and / or other appropriate bit depths). Additionally, high dynamic range (HDR) image data (e.g., HDR10, perceptual quantizer (PQ), etc.) can also be processed. However, in some embodiments, it can be desirable to process or generate statistics from the input image data 50 using a channel that represents luminance values (e.g., a Y channel). The single luminance channel can preserve the content (e.g., edges, angles, lines, etc.) of the image for pixel statistics gathering and interpretation for directional scaling and enhancement. Depending on the color base of the input image data 50, the transform block 56 can generate luminance pixel data that represents a target white point, black point, or gray point of the input image data 50. This luminance pixel data can then be used by the other processing blocks 54. By way of example, if the input image data 50 uses an RGB format, the transform block 56 can apply weighting coefficients to each channel (i.e., a red channel, a green channel, and a blue channel) and combine the results to output a single channel of luminance pixel data. Additionally or alternatively, the processing blocks 54 can use non-luminance pixel data to gather and interpret pixel statistics and for directional scaling and enhancement.

[0063] In one embodiment, the noise statistics block 58 can receive luminance pixel data corresponding to the input image data 50. The noise statistics block 58 can then process the luminance pixel data through one or more vertical and / or horizontal filters and qualify the luminance pixel data corresponding to each pixel. The qualified luminance pixel data can be used to generate noise statistics values from which the noise statistics block 58 can identify patterns in the image data content, for example, for use in the image enhancement block 64. The image enhancement block 64 can take the scaled image data and / or the input image data 50 and enhance (e.g., sharpen) the image data generating enhanced image data using tone detection, comparison between low resolution input and high resolution input, and noise statistics values.

[0064] The angle detection block 60 can also receive luminance pixel data corresponding to the input image data 50. The angle detection block 60 can analyze SAD statistics and DIFF statistics at multiple angles around a pixel of interest to identify angles corresponding to lines and / or edges within the content of the input image data 50. These angles can then be used in the directional scaling block 62 to facilitate improved interpolation of new pixels generated when scaling to a higher resolution (e.g., doubling the size of the original image results in approximately quadrupling the number of pixels). Additionally or alternatively, the vertical / horizontal scaling block 66 can also scale the scaled image data to a higher or lower resolution to match a desired output resolution.

[0065] For ease of illustration, Figure 8 is a flowchart 68 depicting one embodiment of a process performed by the scaler block 46. As described above, the scaler block 46 can transform the input image data 50 to luminance pixel data, e.g., using the transform block 56, if desired (process block 70). The luminance pixel data can be used to determine noise statistics, e.g., via the noise statistics block 58 (process block 72). The luminance pixel data can also be used to determine SAD statistics and / or DIFF statistics (process block 74), which can then be used for angle detection, e.g., using the angle detection block 60 (process block 76). The scaler block 46 can also scale the input image data 50 based at least in part on the detected angles, e.g., via the directional scaling block 62 (process block 78). Using the input image data 50 or the scaled image data, the luminance pixel data and / or the chrominance pixel data can be enhanced, e.g., via the image enhancement block 64, to generate enhanced image data (process block 80). Additionally, if desired, the scaler block 46 can also perform vertical and / or horizontal scaling of the image data, e.g., via the vertical / horizontal scaling block 66 (process block 82).

[0066] As described above, the noise statistics block 58 can take the luminance pixel data 84 and generate noise statistics 86, as shown in Figure 9 The noise statistics 86, which are based at least in part on the content of the input image data 50, can allow noise to be distinguished from more intentional features of the image (e.g., features that are desired to be enhanced). In some embodiments, noise aspects of the image are ignored when undergoing enhancement. To facilitate determining such noise, the noise statistics block 58 can include a pixel eligibility sub-block 88, a vertical filter 90, and / or a horizontal filter 92.

[0067] The luma pixel data 84 can undergo processing in one or more vertical and / or horizontal filters 90 and 92 in series, in sequence, and / or in parallel. Such filters 90 and 92 can include, for example, low pass filters, band pass filters, and / or high pass filters capable of identifying and / or producing frequency content corresponding to different frequency bands of the image. The pixel eligibility sub-block 88 can use the luma pixel data 84 and / or filtered luma pixel data to determine whether the pixel data of each individual pixel is eligible for use in the noise statistics 86. In some embodiments, the noise statistics block 58 can sample every pixel of the luma pixel data 84. However, in some embodiments, the noise statistics block 58 can sample less than the full set of luma pixel data 84 (e.g., one in every 4 pixels), for example, if the directional scaling block 62 is not enabled.

[0068] Determining eligible pixels such that they meet one or more criteria. For example, in one embodiment, if a pixel falls within a valid region, the pixel is eligible for use in the noise statistics 86. In some embodiments, the valid region can be set to group together relevant portions of the image while excluding irrelevant portions of the image. For example, the valid region can be set to exclude sub-titles, constant color sections (e.g., mailboxes), etc. Further, the valid region can be programmable and / or software-optimizable to increase the likelihood of detecting various forms of noise from various content. For example, movie content can include artificial noise, such as film grain, that is intentionally added to each video frame. It can be desirable to use a particular valid region size to detect such artificial noise to increase or decrease enhancement. In some embodiments, the valid region can include the entire image.

[0069] Additionally or alternatively, other criteria can also apply, for example, a percentage of pixels in a window (e.g., a 1x1 (the pixel itself) or 3x3 pixel window) around the pixel of interest can contain luma values within a specified range. Further, a local activity measure (e.g., a sum of filtered or unfiltered luma values of neighboring pixels) can also qualify a pixel for use in the noise statistics 86 if the local activity measure is greater than a threshold.

[0070] Once qualified, the filtered and / or unfiltered luminance pixel data 84 can be used to form noise statistics 86 by determining, for example, a histogram, a sum, a maximum value, a minimum value, a mean value, a variance, and / or a blockiness metric (e.g., a measure of corners and / or edges). The blockiness metric can be indicative of artifacts resulting from, for example, block-based compression techniques. Such noise statistics 86 can represent global luminance values within the effective area and / or local values within a pixel window (e.g., a 5x5 pixel window). The noise statistics 86 can represent a frequency signature (e.g., a frequency band) of the image data indicative of blockiness, noise, or particular features (e.g., film grain, video capture noise, etc.) that can be contained within the image. In some embodiments, features such as film grain are intentional within the image and, thus, are expected to be appropriately enhanced and scaled. Different image features can have different frequency signatures and, thus, can be determined based on an analysis of the programmed frequency bands. The differentiation of such features from noise allows for improved scaling and enhancement of desired image features and reduction of enhancement of noise or noise areas within the image.

[0071] Figure 10 is a flowchart 94 depicting an embodiment of a process for determining noise statistics 86. The noise statistics block 58 can first receive, for example, luminance pixel data 84 from the input image data 50 or as transformed via the transform block 56 (process block 96). The noise statistics block 58 can then apply a vertical and / or horizontal filter to the luminance pixel data (process block 98). The noise statistics block 58 can also determine the qualification of the luminance pixel data (process block 100). The qualified luminance pixel data can populate an update of the noise statistics 86 (process block 102). For example, the noise statistics 86 can be updated by generating a histogram, a maximum value, a sum, a mean value, a variance, a blockiness metric, and / or other suitable metrics of the qualified luminance pixel data (process block 104). The noise statistics block 58 can also differentiate noise and frequency signatures (e.g., preprogrammed frequency bands) of desired content (e.g., film grain) from the content of the image (process block 106). The noise statistics 86 can then be output for use in image enhancement and / or other image processing techniques.

[0072] Similar to the noise statistics block 58, the corner detection block 60 can also take as input the luminance pixel data 84 on which to determine statistics. From such statistics, as Figure 11As depicted, the angle detection block 60 can generate optimal mode data 108. In one embodiment, the optimal mode data 108 can include one or more angles corresponding to lines and edges of the image. Further, the optimal mode data 108 for each sample pixel can include a weight for one or more angles corresponding to a confidence level for the angle and / or a similarity of the angle to those of neighboring pixels. The optimal mode data 108 can facilitate improved directional scaling of the image data. The angle detection block 60 can include an SAD and DIFF statistics collection sub-block 110 with modifiers 112, a classifier 114, a high frequency and low angle detection sub-block 116, an angle consistency and difference setting sub-block 118, and a mode and weight determination sub-block 120. The generation and analysis of SAD and DIFF statistics along with the evaluation of confidence and consistency can yield the optimal mode data 108.

[0073] To generate the SAD and DIFF statistics, the SAD and DIFF statistics collection sub-block 110 can analyze the luminance pixel data 84 in a plurality of directions around each pixel of interest. For example, Figure 12 A plurality of pixel groupings 122 are shown for evaluating the luminance pixel data 84 at different angles. In some embodiments, a rectangular base pixel cluster 124 is used as a reference from which to determine the SAD and DIFF statistics for a pixel of interest. In some embodiments, the pixel of interest in the rectangular base pixel cluster is the upper left pixel, however, other pixel positions can also be used. When compared to the rectangular base pixel cluster 124, offset pixel clusters 126, 128, 130, 132, 134, 136, 138, 140, and 142 can yield information about how the luminance pixel data 84 changes in different directions corresponding to the offset pixel clusters 126, 128, 130, 132, 134, 136, 138, 140, and 142. For example, the offset pixel clusters 126 and 128 can correspond to a 45 degree offset from the rectangular base pixel cluster 124. A 135 degree offset, which is orthogonal to the 45 degree offset, can be represented by the offset pixel clusters 130 and 132. Further, a vertical offset cluster 134 and 136 and a horizontal offset cluster 140 and 142 can also be analyzed. In cases where a pixel cluster includes a pixel position that is not within the active area, the pixel value of the nearest pixel within the active area can be substituted. In one embodiment, the pixel values of pixel positions on the edges of the active area can be repeated horizontally and vertically to define the values of pixels outside of the active area.

[0074] To represent other angles (e.g., angles with slopes that are not 0, -1, 1, or infinite), diagonal base pixel clusters 144, 146, 148, 150, 152, and 154 can be considered, as Figure 13As with the rectangular base pixel cluster 124, the diagonal base pixel clusters 144, 146, 148, 150, 152, and 154 can be shifted by an offset and compared to obtain SAD and DIFF statistics values corresponding to respective angles. In some embodiments, some angles can be better represented by using a greater number of pixels in the pixel cluster. For example, diagonal base pixel cluster 144 can be used when collecting SAD and DIFF statistics values at a 1 / 2 slope, and diagonal base pixel cluster 152 can be used at a 1 / 6 slope. As such, diagonal base pixel cluster 144 can utilize more pixels than rectangular base pixel cluster 124 and less pixels than diagonal base pixel cluster 152.

[0075] The SAD and DIFF statistics collection sub-block 110 can utilize the sum of absolute differences (SAD) between the base and offset pixel groupings 122 to calculate a metric for each desired angle. The angles exhibited by the pixel groupings 122 are shown by way of example and are thus non-limiting. In some embodiments, an evaluation of the content of an image can be done under multiple types of gradients (e.g., slopes, curves, angles, etc.). In addition to using SAD, difference (DIFF) statistics can also be collected. DIFF statistics can include metrics such as the difference between consecutive pixels (e.g., in a line or curve), edge metrics for determining corners and / or edges within the image content, and / or other metrics.

[0076] In some embodiments, it can be desirable to reduce the bit depth of the input image data 50 or other data (e.g., luminance pixel data 84) to a smaller bit depth to facilitate manipulation and / or analysis. Such reduction in bit depth can be achieved through, but not limited to, a shift operation, clamping, clipping, and / or truncation. For example, the bit depth of the data being analyzed can be reduced by a shift operation followed by clamping to reduce resource overhead (e.g., time, computational bandwidth, hardware capabilities, etc.) prior to calculating the SAD and DIFF statistics. Additionally or alternatively, the bit depth reduction can be scalable based on programmable parameters to set a desired amount of bit depth reduction, which can vary depending on the particular implementation (e.g., for high definition image processing or low definition image processing). The bit reduction can result in a bit depth of any granularity, which can or can not be a multiple of two. Further, bit reduction can also be used in other processing blocks 54 to reduce resource overhead.

[0077] The SAD and DIFF statistics collection sub-block 110 can also include a modifier 112 to normalize the angle statistics and account for different numbers of pixels used at different angles. In some implementations, the analysis of each angle and / or metric can be further adjusted by the modifier 112 based on the angle being examined. In some contexts, lower angles (e.g., those with a slope less than 1 / 3 or 1 / 4 or greater than 2 or 3) can be prone to false positives when undergoing SAD and DIFF analysis. As such, the confidence in low angle analysis can be less than the confidence in horizontal or vertical directions, and thus the low angle analysis can be adjusted accordingly, e.g., via the modifier 112. Based on the SAD and DIFF analysis, the classifier 114 can determine one or more best angles. The best angles can correspond to the angles that are closest to the direction of uniformity (e.g., lines, edges, etc.) in the image content. In some implementations, the classifier 114 generates a best angle and a second best angle for further consideration in the angle detection block 60.

[0078] In some implementations, the horizontal and vertical directions can be processed separately from other angles analyzed by the angle detection block 60. For example, depending on the interpolation method during scaling, it can be desirable to interpolate pixels positioned diagonally at angles other than vertical or horizontal. On the other hand, it can also be desirable to use vertical and horizontal interpolation for pixels positioned directly perpendicular or horizontal to the original pixels. As such, in some implementations, the classifier 114 can output the best angle and a second best angle of the non-vertical / horizontal angles as well as the best vertical or horizontal angle.

[0079] After computing one or more best angles, the angle detection block 60 can further evaluate the determined best angles with a high frequency and low angle detection sub-block 116. In some contexts, the content of an image can have high frequency features (e.g., a checkerboard pattern) that can result in inaccurate angle indications of the image (e.g., false angles). The high frequency and low angle detection sub-block 116 can search for such high frequency features, e.g., using the horizontal and vertical DIFF statistics. In some implementations, the best angles can be used to interpolate intermediate pixel values between pixel values of the original pixels, and the high frequency and low angle detection sub-block 116 can check whether the approximate interpolation is consistent with neighboring pixels.

[0080] Additionally or alternatively, the high frequency and low angle detection sub-block 116 can utilize one or more conditions and / or parameters to determine the viability of the determined low angle. For example, the difference between consecutive and / or consecutively considered pixel values (e.g., based on the luminance pixel data 84) can be considered. For a given set of pixels, these difference values (e.g., positive difference values, negative difference values, or zero difference values) can be considered together in a run (e.g., a string of positive difference values, negative difference values, or pixel values with no difference values). The length of such a run can be calculated and used as a parameter of one or more low angle conditions to identify one or more low angle dilemmas. For example, if the run length of pixel value difference values is within a configurable range (e.g., less than and / or greater than a threshold value based on the particular implementation settings), the detected angle can be a false angle (e.g., a false edge detected due to noise), and the confidence of the determined angle can be increased or decreased accordingly.

[0081] In some embodiments, recognizing and calculating the conditions and / or parameters of the high frequency and low angle detection sub-block 116 while maintaining the data throughput of the angle detection block 60 can be expensive to implement in hardware. For example, an 8-bit or 16-bit per pixel value implementation can use eight or sixteen repeated logic circuits, respectively, to determine the run length within a given time period. However, in some embodiments, a logic circuit design that combines forward and backward propagation can provide a single logic circuit that is scalable to multiple different implementations with minimal data path speed cost (e.g., less than 5%, 10%, or 20% per doubling of the bit depth of the luminance pixel data 84) without changing or repeating the logic circuit. As such, the high frequency and low angle detection sub-block 116 can efficiently check the conditions and / or parameters to help identify the confidence of the best angle.

[0082] Additionally, if the best angle and / or the second best angle is a low angle (e.g., a slope less than 1 / 3 and greater than 3) relative to horizontal and vertical and a high frequency feature or a low angle dilemma is detected, the confidence of the low angle can be decreased. In one embodiment, if the best angle is a low angle, the second best angle is not a low angle, and a high frequency feature is detected, the second best angle can be output from the high frequency and low angle detection sub-block 116 as the new best angle, and the old best angle can become the new second best angle.

[0083] Further, the angle detection block 60 can also include an angle consistency and differential setting sub-block 118. In some embodiments, it can be desirable to use the orthogonal angle having the best angle (e.g., for interpolation or comparison). The angle consistency and differential setting sub-block 118 can determine the angle orthogonal to the best angle from the previously analyzed angles. As such, each angle analyzed in the SAD and DIFF statistics collection sub-block 110 can have a counterpart that is also analyzed that is orthogonal or approximately orthogonal. Further, in some embodiments, the best angle and the second best angle can be converted to angle measures and compared to each other. If the difference between the best angle and the second best angle is less than a threshold, they can be considered consistent. Angle consistency can increase the confidence of the best angle and / or decrease the confidence if the angles are not consistent. Further, in some embodiments, the confidence measure of the best angle can also be compared to the confidence measure of its orthogonal angle to further modify the confidence level. For example, if the confidence that a line or edge in the image content exists in the orthogonal direction is almost as high as the confidence of the best angle, the confidence level of the best angle can be decreased.

[0084] The output of the SAD and DIFF statistics collection sub-block 110, the classifier 114, the high frequency and low angle detection sub-block 116, and / or the angle consistency and differential setting sub-block 118 can be fed into a pattern and weight determination sub-block 120. The pattern and weight determination sub-block 120 can determine the best mode data 108 corresponding to the best angle, the best horizontal / vertical angle, and / or the orthogonal angle for each sub-block. Further, the best mode data 108 can include weights based at least in part on the confidence of the angle. Further, the weights given to the best angle and the best horizontal / vertical angle at a particular pixel location can be further based at least in part on the best angle of neighboring pixel locations. For example, if the majority of pixels around a pixel of interest have the same best angle as the pixel of interest, the confidence of the best angle of the pixel of interest can be increased and thus the weight.

[0085] For ease of illustration, Figure 14is a flowchart 156 depicting the operation of the angle detection block 60 for a single pixel location. The angle detection block 60 can first determine absolute difference and statistical values and differential statistical values at a plurality of angles from the luminance pixel data 84 (process block 158). The determined SAD and DIFF statistical values can be normalized / modified, e.g., based on the respective angles (process block 160). Among the analyzed angles, one or more best angles can be determined, e.g., by the classifier 114 (process block 162). Using the best angles, the angle detection block 60 can detect high frequency and low angle occurrences for possible undesirability (process block 164) and adjust the confidence of the angles accordingly. In addition, the angle detection block can determine the angle consistency between the first and second best angles (process block 166) and determine angles orthogonal to the best angles (process block 168). As noted above, the perpendicular and horizontal angles can be processed separately from the rest, and thus can also include best horizontal / vertical angles and corresponding orthogonal angles. The consistency of neighboring pixels with the determined best angles can also be checked (process block 170), e.g., to update the angle confidence. The angle detection block 60 can then output the best mode data 108 including the best angles and corresponding weights (process block 172), e.g., for use in the directional scaling block 62.

[0086] When received by the directional scaling block 62, the best mode data 108 and the input image data 50 can be combined to generate scaled image data 174, as Figure 15 depicted. In one embodiment, the directional scaling block 62 can include a midpoint interpolation sub-block 176 and an outer point interpolation sub-block 178. Although the use of luminance pixel data 84 for angle analysis is stated above, other color channels can also be used to gather statistical values for angle detection and directional scaling. In addition, the best mode data 108 gathered from a single channel can be used to scale multiple color channels. As such, for each color channel, the same weights or derivatives thereof and angles for interpolation can be used in the midpoint interpolation sub-block 176 and the outer point interpolation sub-block 178.

[0087] In some embodiments, a pixel grid 180 can schematically represent the location and relative location of pixels, as Figure 16The directional scaling block 62 can use the input pixels 182 to interpolate the midpoint pixels 184 and the outer point pixels 186. Further, in some embodiments, the midpoint pixels 184 are interpolated before the outer point pixels 186. Because of the lack of pixel data in the vertical or horizontal direction around the midpoint pixels 184, the midpoint pixels 184 can be interpolated in a diagonal manner using the best angle, orthogonal angle, and / or weights from the best mode data 108. Using the neighboring input pixels 182, the midpoint interpolation sub-block 176 can determine a value for each color channel of each midpoint pixel 184. The weighting of the interpolation of each surrounding input pixel 182 is based at least in part on the weights of the best mode data 108. As such, the weights of the best mode data can correspond to the weights in a weighted average of the neighboring pixel values. In some embodiments, a temporary pixel value can be established by interpolating two or more input pixels 182. This temporary pixel value can then be used to interpolate the midpoint pixel 184. Such temporary pixel values can be used to better interpolate the value of the midpoint pixel 184 at a particular angle. Further, in some embodiments, the horizontal / vertical interpolation of the midpoint pixels 184 can be generated based at least in part on the best vertical / horizontal angle and mixed with the diagonal interpolation to generate the value of the midpoint pixel 184.

[0088] Once the midpoint pixels 184 have been determined, the outer point pixels 186 can be determined. Unlike the midpoint pixels 184, each outer point pixel 186 has input or vertically and horizontally determined pixel data around it. As such, the best vertical / horizontal angle and weights can be used to interpolate the outer point pixels 186. For example, if the determined best vertical / horizontal angle is in the vertical direction, the outer point pixel 186 can be interpolated with higher weights given to the pixels above and below the outer point pixel 186. It should be understood that a combination of the vertical / horizontal best angle and the diagonal best angle can also be used for the midpoint pixel interpolation or the outer point interpolation. Further, in some embodiments, the outer point pixels 186 can be determined before the midpoint pixels 184.

[0089] In some embodiments, the directional scaling block 62 can perform fixed rate scaling, such as multiplying the dimensions by 2, 4, etc. To achieve higher or lower levels of resolution scaling, the directional scaling block 62 can be implemented multiple times (e.g., cascaded), and / or the vertical / horizontal scaling block 66 can be used to achieve non-multiple resolutions (e.g., resolutions that are 1.2, 2.5, 3.75, or other multiples of the input resolution). The vertical / horizontal scaling block 66 can include linear scalers, polyphase scalers, and / or any suitable scalers to achieve the desired resolution. Further, scaling can be implemented such that each dimension has a different scaling multiple. Additionally or alternatively, the vertical / horizontal scaling block 66 can scale the input image data 50 in parallel with the directional scaling block 62. In this case, the output of the vertical / horizontal scaling block 66 and the output of the directional scaling block 62 can be blended to generate the scaled image data 174.

[0090] In further illustration, Figure 17 is a flowchart 188 depicting the simplified operation of the directional scaling block 62. The directional scaling block 62 can first receive the input image data 50 and the best mode data 108 (process block 190). The directional scaling block can also diagonally interpolate midpoint pixels 184 between the input pixels 182 (process block 192) and interpolate outer point pixels 186 from the input pixels 182 and the midpoint pixels 184 (process block 194). The scaled image data 174 is then output (process block 196).

[0091] In some embodiments, the scaled image data 174 can be sent to the image enhancement block 64. The image enhancement block 64 can also be used in addition to the scaler block 46. In fact, in some embodiments, the image enhancement block 64 can enhance the input image data 50 without upscaling to a higher resolution. As Figure 18 depicted, the image enhancement block 64 can take the scaled image data 174, the input image data 50, or both, as well as noise statistics as input, and output enhanced image data 198. If the scaled image data 174 is not available, the image enhancement block 64 can subsample the input image data 50 (e.g., 1 out of 4 pixels) using a down-sampling sub-block 200. This can also correspond to the subsampling of the noise statistics block 58, where the subsample of the input image data 50 can be used if the directional scaling block 62 is disabled. The down-sampled image data can be used as a low resolution input for the example-based refinement. If the scaled image data 174 is available, the input image data 50 can be used as the low resolution input.

[0092] The image enhancement block 64 can also include a hue detection sub-block 202, a luminance processing sub-block 204, and a chrominance processing sub-block 206. The hue detection sub-block 202 can search the image content for recognizable hues that match possible image representations (e.g., sky, grass, skin, etc.). In some embodiments, the hue detection sub-block 202 can combine multiple color channels to determine whether a recognizable hue is present. Also, in some embodiments, the hue detection sub-block 202 can convert one or more color channels to a hue, saturation, purity (HSV) format for analysis. Each searched for hue can be given a confidence level based at least in part on the likelihood that the detected hue is characteristic of an image representation. By including an improved assessment of the image content, the recognition of hues within the image can result in improved enhancement of the image. For example, the luminance processing sub-block 204 and the chrominance processing sub-block 206 can use the hue data to adjust (e.g., with increased or decreased enhancement) the luminance values and chrominance values of the input image data 50 or the scaled image data 174 in areas where a hue was detected. In one embodiment, the effect on the areas of various hues can be software programmable.

[0093] In one embodiment, the luminance processing sub-block 204 enhances (e.g., sharpens) the luminance channel of the input image data 50 or the scaled image data 174 and includes a luminance transition improvement 208 and an example-based improvement 210. The luminance transition improvement 208 can include one or more horizontal or vertical filters (e.g., high pass and / or low pass) arranged with adaptive or programmable gain logic as a peak filter. The peak filter can boost a programmable range of frequencies corresponding to content features of the image (e.g., crosshatching, other high frequency components). The boosted frequency range can provide better frequency and / or spatial resolution to the luminance channel. Also, the luminance transition improvement 208 can include a de- core circuit to minimize the amount of luminance enhancement in noisy regions of the input image data 50, for example, as determined by the noise statistics block 58. Further, the luminance transition improvement 208 can use an edge metric (e.g., the edge metric determined from the SAD and DIFF statistics values collection sub-block 110) within the de-core circuit to reduce overshoot and / or undershoot that can occur near edge transitions, for example, due to the boosted frequency range.

[0094] Further, the example-based improvement 210 can operate in parallel or in series with the luminance transition improvement 208 as part of the luminance processing sub-block 204. The example-based improvement 210 can take a low resolution input 214 and compare segments (e.g., 5x5 pixel segments) thereof to segments of a high resolution input 216 (e.g., the input image data 50 or the scaled image data 174), as shown in FIG. 2. The example-based improvement 210 can use the comparison to determine a confidence level for each segment of the low resolution input 214. The example-based improvement 210 can then use the confidence levels to adjust the luminance values of the input image data 50 or the scaled image data 174 in the areas of the low resolution input 214 where a segment was detected. Figure 19The example-based refinement 210 can gather a plurality (e.g., 25) of segments (e.g., 5x5 pixel segments) of the low resolution input 214 and compare each segment to a single segment of the high resolution input 216, in some embodiments. In addition, the low resolution input 214 can pass through a filter 218 (e.g., a low pass filter) to generate a filtered low resolution input 220. The high resolution input 216, the low resolution input 214, and / or the filtered low resolution input 220 can be evaluated in a comparison and weight sub-block 222. For example, the comparison and weight sub-block 222 can utilize a sum of squared differences or a squared difference approximation of luminance channel values. In some embodiments, employing a sum of squared differences can be a resource (e.g., time, computational bandwidth) intensive process, and thus it can be desirable to instead utilize a squared difference approximation.

[0095] A squared difference approximation can be implemented between each value of a segment of the low resolution input 214 and each value of a segment of the high resolution input 216. In one embodiment, a single squared difference can be estimated using a function that returns the number of leading zeros of a bit value corresponding to a difference between a value of a segment of the high resolution input 216 and a corresponding value of a segment of the low resolution input 214. A sum of the squared difference approximations can then at least partially represent an approximation of a sum of squared differences between the segment of the high resolution input 216 and the segment of the low resolution input 214. The squared difference approximation can be implemented between each of a plurality (e.g., 25) of segments (e.g., 5x5 pixel segments) of the low resolution input 214 and a single segment of the high resolution input 216 for use in the comparison and weight sub-block 222. Other operations, such as clipping, multiplication by a programmable parameter, bit shifting, etc., can also be included in the calculation of the squared difference approximation or the sum of squared differences approximation.

[0096] From the similarities and differences of the high resolution input 216, the low resolution input 214, and / or the filtered low resolution input 220, the comparison and weight sub-block 222 can determine a weight from which a weighted average of the inputs is generated. For example, the comparison and weight sub-block 222 can apply a lookup table to the similarities and / or differences to generate a weight for the weighted average. Based at least in part on the generated weight, the blending sub-block 224 can combine the inputs to generate refined luminance data 226. In addition, the luminance processing sub-block 204 can combine the refined luminance data 226 from the example-based refinement 210 with the peak and core refinement of the luminance transition refinement 208 based on, for example, gradient statistics.

[0097] Gradient statistics can indicate linear changes in pixel values in a particular direction, e.g., in the x-direction and / or y-direction relative to the pixel grid 180. For example, a weighted average of changes in pixel values in the x-direction can be combined with a weighted average of changes in pixel values in the y-direction to determine how to mix improved luminance data 226 from the example-based improvement 210 with the peak and core improvements of the luminance transition improvement 208. The example-based improvement 210 can produce improved identification and display of dominant gradients within an image, and the luminance transition improvement 208 can improve perceived texture in the image, and the combination of the two can allow for an enhanced (e.g., sharpened) luminance channel output.

[0098] Similar to the luminance transition improvement 208, the chrominance processing sub-block 206 can include a chrominance transition improvement 212 that includes a peak filter and a core circuit. In some embodiments, the chrominance transition improvement 212 can be further enhanced based at least in part on the luminance transition improvement 208. In some contexts, if the luminance channel is enhanced without compensating for the chrominance channel, the image can appear over-saturated or under-saturated. As such, the chrominance transition improvement 212 can employ luminance channel changes due to the luminance processing sub-block 204 when determining changes from the chrominance transition improvement 212. Additionally or alternatively, the chrominance transition improvement 212 can be disabled if, for example, there is little or no luminance channel enhancement. As output from the image enhancement block 64, the enhanced image data 198 (e.g., the enhanced luminance and chrominance channels) can represent a sharpened and vivid image.

[0099] To facilitate further explanation, Figure 20 is a flowchart 228 representing an example process of the image enhancement block 64. The image enhancement block 64 can receive input image data 50 (process block 230) and determine whether scaled image data 174 is available (decision block 232). If the scaled image data 174 is not available, the input image data 50 can be down-sampled for use as a low resolution input 214 for the example-based improvement 210 (process block 234). However, if the scaled image data 174 is available, the scaled image data 174 can be received (process block 236), and the input image data 50 can be used as a low resolution input 214 for the example-based improvement 210 (process block 238). As such, the example-based improvement 210 can be determined (process block 240). Further, the image enhancement block 64 can determine tone detection, e.g., via the tone detection sub-block 202 (process block 242). The image enhancement block 64 can also determine the luminance transition improvement 208 (process block 244) and a luminance channel output (process block 246). The chrominance transition improvement 212 can also be determined, e.g., using the luminance channel output (process block 248), and a chrominance channel output can be determined (process block 250). The luminance channel output and the chrominance channel output together form the enhanced image data 198.

[0100] In some embodiments, the enhanced image data 198 can be scaled after enhancement. For example, the enhanced image data 198 can pass through a vertical / horizontal scaling block 66 after enhancement. Additionally or alternatively, the enhanced image data 198 can be scaled in a directional scaling block 62 before and / or after enhancement. Scaling and / or enhancement can be cascaded multiple times until a desired resolution is achieved.

[0101] It should be appreciated that the various components of the scaler block 46 (e.g., the transform block 56, the noise statistics block 58, the angle detection block 60, the directional scaling block 62, the image enhancement block 64, the vertical / horizontal scaling block 66) can be enabled, disabled, or employed together or separately depending on the implementation. Further, the order of use within the scaler block 46 can also be altered (e.g., switched, repeated, run in parallel or in series, etc.) depending on the implementation. As such, although the flowchart referenced above shows a given order of process / decision blocks, in some embodiments, the process / decision blocks can be reordered, altered, deleted, and / or occur simultaneously. Further, the referenced flowchart is given as an illustrative tool, and additional decision and process blocks can also be added depending on the implementation.

[0102] The foregoing detailed description has shown, by way of example, various embodiments of the described specific implementations. It will be clear to those skilled in the art, however, that various modifications and alternative forms can be made to the specific implementations described without departing from the spirit and scope of the disclosure. Accordingly, the claims are not intended to be limited to the particular forms disclosed, but are to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0103] The technology described and claimed herein was made with U.S. Government support and is subject to the provisions of 37 C.F.R. § 1.701-1.703. The technology described and claimed herein is referenced and applied to specific examples of actual items and actual nature, which significantly improve the art, and thus is not abstract, intangible or purely theoretical. Further, if any of the claims appended to this specification contain one or more dependent claims followed by a comma and the words “claiming the benefit of” or the like, then it is the intent of the Applicant(s) that such dependent claims be interpreted affording broadest scope of the claims under 35 U.S.C. § 112(f), following the en banc decision of the United States Court of Appeals for the Federal Circuit in the case of In re Boehringer Ingelheim, 557 F.3d 1330, 1340, 90 USPQ2d 1869, 1877 (Fed. Cir. 2009). However, for any claim containing elements designated in any other manner, such elements should not be interpreted under 35 U.S.C. § 112(f).

Claims

1. An electronic device including enhancement circuitry configured to enhance high-resolution image data, thereby improving the perceptual quality of an image corresponding to the high-resolution image data, wherein the enhancement circuitry includes: A tone detection circuit configured to determine one or more tones within the image and apply a first change to the high-resolution image data based at least in part on the one or more tones; An improved circuit based on the example is configured to compare the high-resolution image data with low-resolution image data and apply a second change to the high-resolution image data based at least in part on the difference between segments of the high-resolution image data and segments of the low-resolution image data. as well as A channel processing circuit configured to apply a first change and a second change to one or more channels of the high-resolution image data.

2. The electronic device of claim 1, wherein the one or more hues include hues representing at least one of skin, sky, and grass.

3. The electronic device according to claim 1, wherein the channel processing circuit includes a luminance processing circuit and a chrominance processing circuit.

4. The electronic device of claim 3, wherein the brightness processing circuit includes a brightness transition improvement, the brightness transition improvement including an increase in the frequency range within the brightness channel of the high-resolution image data.

5. The electronic device of claim 4, wherein the improvement in the frequency range is based at least in part on the output of the peak filter, wherein the frequency range corresponds to a high-frequency pattern.

6. The electronic device of claim 5, wherein the high-frequency pattern includes crosshairs.

7. The electronic device of claim 4, wherein the example-based improved circuitry operates in parallel with the brightness transition improvement.

8. The electronic device of claim 3, wherein the chroma processing circuit is configured to enhance the chroma channel at least in part based on the enhancement of the luminance channel by the luminance processing circuit.

9. The electronic device of claim 1, further comprising a downsampling circuit configured to generate the low-resolution image data by sampling a portion of the high-resolution image data.

10. The electronic device of claim 9, wherein the portion of the high-resolution image data comprises one of every four pixel values ​​of the high-resolution image data.

11. The electronic device of claim 1, wherein the high-resolution image data is generated by scaling the low-resolution image data to a higher resolution.

12. The electronic device of claim 1, wherein the enhancement circuit is configured to: Receive the high-resolution image data, wherein the high-resolution image data includes a high-resolution luminance channel and a low-resolution luminance channel corresponding to the image, wherein comparing the high-resolution image data with the low-resolution image data includes comparing the high-resolution luminance channel with the low-resolution luminance channel to generate the difference value; The second change is generated at least in part based on the difference; A third modification to the high-resolution luminance channel is generated, wherein the third modification is based at least in part on the following: Improvement of the frequency range of the high-resolution luminance channel; as well as Minimize the correction for undershoot or overshoot at the edges of the image; and The high-resolution luminance channel is enhanced at least in part based on a combination of the second and third changes.

13. A method comprising: The high-resolution luminance channel and low-resolution luminance channel corresponding to the image are received via an enhancement circuit. The high-resolution luminance channel is compared with the low-resolution luminance channel via the enhancement circuit to generate a difference. The enhancement circuit generates a first change to the high-resolution luminance channel based at least in part on the difference; A second change to the high-resolution luminance channel is generated via the enhancement circuitry, wherein the second change is based at least in part on the following: Improvement of the frequency range of the high-resolution luminance channel; as well as Minimize the correction of undershoot or overshoot at the edges of the image; The high-resolution luminance channel is enhanced by the enhancement circuitry, at least in part based on a mixture of the first change and the second change, thereby generating an enhanced luminance channel. as well as Output the enhanced luminance channel.

14. The method of claim 13, wherein the difference is determined at least in part based on a squared difference approximation, wherein the first change to the high-resolution luminance channel is determined by applying a lookup table to the difference.

15. The method of claim 13, further comprising reducing the amount of enhancement to said portion of the high-resolution luminance channel via said enhancement circuitry, at least in part based on determining that a portion of the high-resolution luminance channel contains noise.

16. The method of claim 13, further comprising receiving the chroma channel via the enhancement circuit and enhancing the chroma channel at least in part based on the enhanced luminance channel.

17. A system comprising: Processor, the processor being configured to: Receive high-resolution image data corresponding to the image; Determine one or more hues within the image, and apply a first change to the high-resolution image data based at least in part on the one or more hues; The high-resolution image data is compared with low-resolution image data, and a second modification is applied to the high-resolution image data based at least in part on the difference between segments of the high-resolution image data and segments of the low-resolution image data; as well as A first modification and a second modification are applied to one or more channels of the high-resolution image data to enhance the high-resolution image data, thereby improving the perceptual quality of the image to generate enhanced image data; as well as A controller configured to acquire the high-resolution image data from a memory used by the processor and output the enhanced image data to an electronic display or the memory.

18. The system of claim 17, wherein the processor is configured to generate the low-resolution image data by downsampling the high-resolution image data.

19. The system of claim 17, wherein the processor is configured to limit enhancement in regions of the image based at least in part on noise statistics.

20. The system of claim 17, wherein the processor is configured to compare the high-resolution image data with the low-resolution image data via an example-based improvement.

Images