Directional zoom system and method

Abstract

The present application relates to directional scaling systems and methods. An electronic device may include scaling circuitry to scale input pixel data to a greater resolution. The directional scaling circuitry may include a first interpolation circuit for receiving optimal mode data, the optimal mode data including one or more angles corresponding to the content of an image, and interpolating a first pixel value at a first pixel position diagonally offset from an input pixel position of the input pixel data based on the optimal mode data and an input pixel value corresponding to the input pixel position. The directional scaling circuitry may also include a second interpolation circuit for receiving the optimal mode data and the input pixel value, and interpolating a second pixel value at a second pixel position horizontally or vertically offset from the input pixel position based at least in part on the optimal mode data and the input pixel value.

Description

[0001] This application is a divisional application of the invention patent application with the application date of July 24, 2019, application number 201980051423.7, and name “Directional Zooming System and Method”. Background Art

[0002] The present disclosure relates generally to image processing and, more particularly, to analysis of pixel statistics, scaling, and / or enhancement of image data for displaying images on electronic displays.

[0003] This section is intended to introduce the reader to various aspects of the art that may be related to various aspects of the present disclosure, which are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements should be read in this light, and not as admissions of prior art.

[0004] Electronic devices typically use one or more electronic displays to present visual representations of information, such as text, static images, and / or video, by displaying one or more images (e.g., image frames). For example, such electronic devices may include computers, mobile phones, portable media devices, tablet computers, televisions, virtual reality headsets, and vehicle dashboards. To display images, an electronic display may control the light emission (e.g., brightness) of its display pixels based at least in part on corresponding image / pixel data.

[0005] Generally speaking, image data may indicate a resolution corresponding to the image (e.g., the size of the pixels to be used). However, in some cases, it may be desirable to scale the image to a higher resolution, such as for display on an electronic display with a higher resolution output. Therefore, before the image data is used to display the image, it may be processed to convert the image data to the desired resolution. However, at least in some cases, the techniques used to scale the image data may affect the perceived image quality of the corresponding image, for example, by introducing image artifacts such as jagged edges. When undergoing image enhancement, pixel statistics may be employed to correct for such artifacts. Summary of the Invention

[0006] The following describes a summary of certain embodiments disclosed herein. It should be understood that these aspects are presented merely to provide the reader with a concise summary of these specific embodiments, and that these aspects are not intended to limit the scope of the present disclosure. In fact, the present disclosure may encompass a number of aspects that may not be described below.

[0007] In some cases, the electronic device may scale and / or enhance the image data to improve the perceived quality of the image. In some embodiments, the change to the image data may be based at least in part on the content of the image corresponding to the image data. Such changed image data may be stored in a memory or displayed on an electronic display. In some embodiments, the image data may indicate a target brightness per color component (e.g., channel) via, for example, red component image data, blue component image data, and green component image data. Additionally or alternatively, the image data may indicate a target brightness in grayscale (e.g., grayscale) or via luminance and chrominance components (e.g., YCbCr).

[0008] To facilitate image data improvement, noise statistics can be collected and analyzed. For example, frequency bands within the image data can be identified to help distinguish image content from noise. This can enhance image content while minimizing the impact of noise on the output image. In some embodiments, pixels that do not meet certain criteria can be excluded from the noise statistics.

[0009] In addition, difference statistics and sum of absolute differences (SAD) can be applied to pixel groups of the image data. Such pixel groups can be selected and compared in multiple directions relative to the pixel of interest via angle detection circuitry to identify an optimal pattern (e.g., angle) for interpolation. Comparison of different pixel groups can identify features (edges, lines, and / or changes) in the image content that can assist in using the optimal pattern to enhance or scale the image data. The optimal pattern data can include one or more angles that most accurately describe features of the image content at the location of the pixel of interest. In addition, optimal pattern data from multiple pixels of interest can be compiled together for use in pixel value interpolation.

[0010] For example, in one embodiment, the differential statistics and the SAD statistics can facilitate an increase in image resolution by interpolating new pixel values based, at least in part, on the best mode data. The directional scaling circuit can utilize the identified angle to maintain the features of the image while minimizing the introduction of artifacts such as jagged edges (e.g., aliasing). In some embodiments, the directional scaling circuit can interpolate pixels that are diagonal to the original image data pixel positions by generating a weighted average of the original pixels. The weighted average can, for example, be based on the differential statistics and the SAD statistics and the angle identified therefrom. In addition, the directional scaling circuit can generate horizontally positioned and vertically positioned pixel values from the original pixels by determining a weighted average of the original pixels and / or new diagonal pixels. The weighted average can also be based, at least in part, on the differential statistics and the SAD statistics.

[0011] In conjunction with or separately from the directional scaling circuit, the enhancement circuit may also use differential statistics and SAD statistics to adjust the image data of the image. Additionally or alternatively, the enhancement circuit may use noise statistics and / or a low-resolution version of the image to generate image enhancements. Such enhancements may provide increased sharpness to the image. In some embodiments, example-based enhancement using a lower-resolution version of the image for comparison may provide enhancements to one or more channels of the image data. For example, the luminance channel of the image data may be enhanced based on the sum of squared differences between the image and the low-resolution version of the image, or an approximation thereof. Furthermore, the enhancement circuit may employ a peaking filter to enhance high-frequency aspects of the image (e.g., cross-hatching). Such enhancements may provide improved spatial resolution and / or reduced blur. Furthermore, the enhancement circuit may determine hues within the image to identify certain content (e.g., sky, grass, and / or skin). Example-based enhancement, peaking filters, and / or hue determination may each target different textures of the image to combine the enhancements, and the enhancements derived from each of them may be controlled independently and based on local features of the image.

[0012] Depending on the specific implementation, the noise statistics circuitry, angle detection circuitry, directional scaling circuitry, and enhancement circuitry may be used individually and / or in combination to facilitate improved perceived image quality and / or alter image data to a higher resolution while reducing the likelihood of image artifacts. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Various aspects of the present disclosure may be better understood upon reading the following detailed description and referring to the accompanying drawings, in which:

[0014] Figure 1 is a block diagram of an electronic device including an electronic display according to an embodiment;

[0015] Figure 2 According to the implementation plan Figure 1 Examples of electronic devices;

[0016] Figure 3 According to the implementation plan Figure 1 Another example of an electronic device;

[0017] Figure 4 According to the implementation plan Figure 1 Another example of an electronic device;

[0018] Figure 5 According to the implementation plan Figure 1 Another example of an electronic device;

[0019] Figure 6 is coupled to the Figure 1 A block diagram of a memory processing pipeline of an electronic device;

[0020] Figure 7 According to the implementation plan, Figure 1 A block diagram of a scaler block used in an electronic device;

[0021] Figure 8 is used for operation according to the implementation scheme Figure 7 A flowchart of the process of the scaler block;

[0022] Figure 9 According to the implementation plan Figure 7 A block diagram of the noise statistics block implemented in the scaler block of FIG.

[0023] Figure 10 is used for operation according to the implementation scheme Figure 9 Flowchart of the process of the noise statistics block;

[0024] Figure 11 According to the implementation plan Figure 7 A block diagram of an angle detection block implemented in a scaler block of FIG.

[0025] Figure 12 is a schematic diagram of pixel locations and exemplary sampling thereof according to an embodiment;

[0026] Figure 13 is a schematic diagram of pixel locations and exemplary sampling thereof according to an embodiment;

[0027] Figure 14 is used for operation according to the implementation scheme Figure 11 A flowchart of the process of the angle detection block;

[0028] Figure 15 According to the implementation plan Figure 7 A block diagram of a directional scaler block implemented in a scaler block of ;

[0029] Figure 16 is a schematic diagram of exemplary pixel interpolation points according to an embodiment;

[0030] Figure 17 is used for operation according to the implementation scheme Figure 15 a flow chart of the process of the directional calibrator block;

[0031] Figure 18 is Figure 7 A block diagram of an image enhancement block implemented in a scaler block of FIG.

[0032] Figure 19 is Figure 18 a block diagram of example-based improvements implemented in an image enhancement block of ; and

[0033] Figure 20 is used for operation according to the implementation scheme Figure 18 Flowchart of the process of the enhancement block. DETAILED DESCRIPTION

[0034] One or more specific embodiments will be described below. In order to provide a brief description of these embodiments, not all features of an actual implementation are described in this specification. It should be understood that in the development of any such actual implementation, as in any engineering or design project, many implementation-specific decisions must be made to achieve the developer's specific goals, such as meeting system-related and business-related constraints that may vary from one implementation to another. In addition, it should be understood that such development work may be complex and time-consuming, but it will still be a routine task of design, processing, and manufacturing for those of ordinary skill in the art who benefit from this disclosure.

[0035] To facilitate conveying information, electronic devices typically use one or more electronic displays to present visual representations of information via one or more images (e.g., image frames). Such electronic devices may include computers, mobile phones, portable media devices, tablet computers, televisions, virtual reality headsets, and vehicle dashboards, among others. Additionally or alternatively, the electronic display may take the form of a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a plasma display, and the like.

[0036] In any case, to display an image, an electronic display typically controls the light emission (e.g., brightness and / or color) of its display pixels based on corresponding image data received at a specific resolution (e.g., pixel size). For example, an image data source (e.g., a memory, an input / output (I / O) port, and / or a communication network) may output image data as a stream of pixel data (e.g., image data), wherein the data for each pixel indicates a target luminance (e.g., brightness and / or color) for one or more display pixels located at the corresponding pixel location. In some embodiments, the image data may indicate the luminance of each color component, for example, via red component image data, blue component image data, and green component image data (collectively referred to as RGB). Additionally or alternatively, the image data may be indicated by a luminance channel and one or more chrominance channels (e.g., YCbCr, YUV, etc.), grayscale (e.g., grayscale), or other color basis. It should be understood that, as disclosed herein, a luminance channel may encompass linear luminance values, nonlinear luminance values, and / or gamma-corrected luminance values.

[0037] To facilitate improved perceived image quality, image data may be processed before being output to an electronic display or stored in memory for later use. For example, a processing pipeline implemented via hardware (e.g., circuitry) and / or software (e.g., execution of instructions stored in a tangible, non-transitory medium) may facilitate such image processing. In some cases, it may be desirable to scale the image data to a higher resolution, e.g., to match the resolution of the electronic display or to make the image appear larger. However, in at least some cases, this may affect perceived image quality, for example, by causing perceptible visual artifacts such as blurring, jagged edges (e.g., aliasing), and / or loss of detail.

[0038] Therefore, in order to promote improved perceived image quality, the present disclosure provides techniques for identifying the content of an image (e.g., via statistics), scaling image data to increase resolution while maintaining image clarity (e.g., sharpness), and / or enhancing image data to increase image clarity. In some embodiments, the processing pipeline may include a scaler block to directionally scale image data while taking into account lines, edges, patterns, and angles within the image. Such content-dependent processing may allow image data to be scaled to a higher resolution without artifacts or with a reduced amount of artifacts. In one embodiment, the ability to increase image resolution without introducing noticeable artifacts may allow images to be stored at a lower resolution, thereby saving memory space and / or bandwidth, and restoring the image to a higher resolution before displaying the image. In addition, the image data may undergo further enhancement before being output or stored.

[0039] To address this type of content-dependent processing, the scaler block may include, for example, a noise statistics block, an angle detection block, a directional scaling block, and an image enhancement block. The noise statistics block may use statistical analysis of collected pixel statistics to identify noise and distinguish it from the rest of the image data. In this way, if / when the image data undergoes enhancement, the noise may be ignored or given less weight. The angle detection block may collect statistics based on sum of absolute differences (SAD) and / or differences (DIFF). These SAD statistics and DIFF statistics may be determined at multiple angles around the input pixel to identify angles of interest from which basic scaling interpolation and / or enhancement may be performed. Thus, the best mode data identified for each input pixel (including, for example, best angle, weight, etc.) may help characterize the image content to facilitate directional scaling of the image data. The directional scaling block may obtain input image data and best mode data, and interpolate midpoint pixels and outer point pixels to generate new pixel data to add to the input image data, thereby generating scaled image data. The scaled image data may further be enhanced via the image enhancement block by identifying tones within the image and by comparing the scaled image data to the input image data via example-based refinement. Thus, the scaler block may incorporate hardware components and / or software components to facilitate determining noise and angles of interest, scaling image data to a higher resolution while reducing the likelihood of image artifacts, and / or image enhancement.

[0040] To help illustrate, Figure 1 An electronic device 10 is shown that may include an electronic display 12. As will be described in more detail below, the electronic device 10 may be any suitable electronic device 10, such as a computer, a mobile phone, a portable media device, a tablet computer, a television, a virtual reality headset, a vehicle dashboard, etc. Therefore, it should be noted that Figure 1 It is merely one example of a particular implementation and is intended to illustrate the types of components that may be present in electronic device 10 .

[0041] In the depicted embodiment, the electronic device 10 includes an electronic display 12, one or more input devices 14, one or more input / output (I / O) ports 16, a processor core complex 18 having one or more processors or processor cores, a local memory 20, a main memory storage device 22, a network interface 24, a power supply 26, and image processing circuitry 27. Figure 1The various components described in the accompanying drawings may include hardware elements (e.g., circuitry), software elements (e.g., tangible, non-transitory computer-readable media storing instructions), or a combination of hardware and software elements. It should be noted that the various depicted components may be combined into fewer components or separated into additional components. For example, local memory 20 and main memory storage device 22 may be included in a single component. Additionally or alternatively, image processing circuitry 27 (e.g., a graphics processing unit) may be included in processor core complex 18.

[0042] As shown, the processor core complex 18 is operatively coupled to a local memory 20 and a main memory storage device 22. Thus, the processor core complex 18 can execute instructions stored in the local memory 20 and / or the main memory storage device 22 to perform operations such as generating and / or transmitting image data. As such, the processor core complex 18 can include one or more general-purpose microprocessors, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), or any combination thereof.

[0043] In addition to instructions, the local memory 20 and / or the main memory storage device 22 may store data to be processed by the processor core complex 18. Thus, in some embodiments, the local memory 20 and / or the main memory storage device 22 may include one or more tangible, non-transitory computer-readable media. For example, the local memory 20 may include random access memory (RAM), and the main memory storage device 22 may include read-only memory (ROM), rewritable non-volatile memory (such as flash memory, a hard drive, an optical disk, etc.).

[0044] As shown, processor core complex 18 is also operably coupled to network interface 24. In some embodiments, network interface 24 can facilitate data communication with another electronic device and / or a communication network. For example, network interface 24 (e.g., a radio frequency system) can enable electronic device 10 to communicatively couple to a personal area network (PAN), such as a Bluetooth network, a local area network (LAN) (such as an 802.11x Wi-Fi network), and / or a wide area network (WAN) (such as a 4G or LTE cellular network).

[0045] Furthermore, as shown, the processor core complex 18 may be operatively coupled to a power source 26. In some embodiments, the power source 26 may provide power to one or more components in the electronic device 10, such as the processor core complex 18 and / or the electronic display 12. Thus, the power source 26 may include any suitable energy source, such as a rechargeable lithium polymer (Li-poly) battery and / or an alternating current (AC) power converter.

[0046] In addition, as shown, the processor core complex 18 is operatively coupled to one or more I / O ports 16. In some embodiments, the I / O ports 16 can enable the electronic device 10 to interface with other electronic devices. For example, when a portable storage device is connected, the I / O ports 16 can enable the processor core complex 18 to communicate data with the portable storage device.

[0047] As shown, electronic device 10 is also operably coupled to one or more input devices 14. In some embodiments, input devices 14 can facilitate user interaction with electronic device 10 by, for example, receiving user input. Thus, input devices 14 can include buttons, a keyboard, a mouse, a trackpad, and the like. Additionally, in some embodiments, input devices 14 can include touch-sensing components within electronic display 12. In such embodiments, the touch-sensing components can receive user input by detecting the occurrence and / or location of an object touching the surface of electronic display 12.

[0048] In addition to enabling user input, electronic display 12 may include a display panel having one or more display pixels. Electronic display 12 may control light emission from its display pixels to present a visual representation of information, such as an operating system graphical user interface (GUI), an application interface, a static image, or video content, by displaying a frame based at least in part on corresponding image data (e.g., image pixel data at each pixel location).

[0049] As shown, the electronic display 12 is operatively coupled to the processor core complex 18 and the image processing circuitry 27. In this manner, the electronic display 12 can display an image based, at least in part, on image data received from an image data source (such as the processor core complex 18 and / or the image processing circuitry 27). In some embodiments, the image data source can generate source image data to create a digital representation of the image to be displayed. In other words, the image data is generated so that the view on the electronic display 12 accurately represents the intended image. Additionally or alternatively, the electronic display 12 can display an image based, at least in part, on image data received via the network interface 24, the input device 14, and / or the I / O port 16. To facilitate accurate image representation, the image data can be processed, for example, via a processing pipeline and / or display pipeline implemented in the processor core complex 18 and / or the image processing circuitry 27, before being supplied to the electronic display 12. Furthermore, in some embodiments, the image data can be obtained, for example, from the memory 20, processed, for example, in the processing pipeline, and returned to its source (e.g., the memory 20). As described herein, such techniques are referred to as memory-to-memory processing.

[0050] As will be described in greater detail below, the processing pipeline may perform various processing operations, such as image scaling, rotation, enhancement, pixel statistics collection and interpretation, spatial and / or temporal dithering, pixel color space conversion, brightness determination, brightness optimization, etc. For example, the processing pipeline may directionally scale the image data to increase resolution while using the content of the image data to reduce the likelihood of producing perceptible visual artifacts (e.g., jagged edges, banding, and / or blurring) when displaying the corresponding image on the electronic display 12.

[0051] In some embodiments, after receiving the image data, the electronic display 12 may perform additional processing on the image data, e.g., to further improve the accuracy of the image being viewed. For example, the electronic display 12 may further scale, rotate, spatially dither, and / or enhance the image data. Thus, in some embodiments, a processing pipeline may be implemented using the electronic display 12.

[0052] As mentioned above, the electronic device 10 can be any suitable electronic device. For ease of illustration, an example of a suitable electronic device 10, particularly a handheld device 10A, is shown in FIG. Figure 2 In some embodiments, the handheld device 10A may be a portable phone, a media player, a personal data organizer, a handheld gaming platform, or the like. For purposes of illustration, the handheld device 10A may be a smartphone, such as any available from Apple Inc. model.

[0053] As shown, handheld device 10A includes a housing 28 (e.g., a casing). In some embodiments, housing 28 can protect internal components from physical damage and / or shield internal components from electromagnetic interference. Additionally, as shown, housing 28 can surround electronic display 12. In the depicted embodiment, electronic display 12 displays a graphical user interface (GUI) 30 having an array of icons 32. For example, when an icon 32 is selected via input device 14 or a touch-sensitive component of electronic display 12, an application can be launched.

[0054] In addition, as shown, input device 14 can be accessed through the opening in housing 28. As mentioned above, input device 14 can enable a user to interact with handheld device 10A. For example, input device 14 can enable a user to activate or deactivate handheld device 10A, navigate the user interface to the home screen, navigate the user interface to a user-configurable application screen, activate a voice recognition feature structure, provide volume control and / or switch between vibration and ring mode. As shown, I / O port 16 can be accessed through the opening in housing 28. In some embodiments, I / O port 16 may include, for example, an audio jack that is connected to an external device.

[0055] For further explanation, another example of a suitable electronic device 10, particularly a tablet device 10B, is shown in FIG. Figure 3 For illustrative purposes, tablet device 10B may be any device commercially available from Apple Inc. Model No. Figure 4 Another example of a suitable electronic device 10 is shown in FIG. 1 , which is specifically a computer 10C. For illustrative purposes, the computer 10C may be any computer available from Apple Inc. or Another example of a suitable electronic device 10, particularly a watch 10D, is shown in Figure 5 For illustrative purposes, the watch 10D may be any Apple As shown, tablet device 10B, computer 10C, and watch 10D each further include an electronic display 12 , an input device 14 , an I / O port 16 , and a housing 28 .

[0056] As described above, the electronic display 12 may display an image based on image data received from an image data source. Figure 6 A portion 34 of the electronic device 10 is shown that includes a processing pipeline 36 that operatively retrieves, processes, and outputs image data. In some embodiments, the processing pipeline 36 may analyze and / or process image data received from the memory 20, for example, by directionally scaling and enhancing the image data before the image data is used to display an image or is stored in the memory 20, as in memory-to-memory processing. In this scenario, the image data may be directionally scaled to a higher resolution and then stored in the memory for later viewing. In some embodiments, the processing pipeline 36 may be incorporated into or coupled to the display pipeline and, thus, be operatively coupled to the display driver 38 to generate analog and / or digital electrical signals based at least in part on the image data and supply them to the display pixels of the electronic display 12.

[0057] In some embodiments, processing pipeline 36 may be implemented in electronic device 10, electronic display 12, or a combination thereof. For example, processing pipeline 36 may be included in processor core complex 18, image processing circuitry 27, a timing controller (TCON) in electronic display 12, one or more other processing units or circuits, or any combination thereof.

[0058] In some embodiments, the controller 40 can control the operation of the processing pipeline 36, the memory 20, and / or the display driver 38. To facilitate the control operations, the controller 40 can include a controller processor and a controller memory. In some embodiments, the controller processor can execute instructions stored in the controller memory, such as firmware. In some embodiments, the controller processor can be included in the processor core complex 18, the image processing circuit 27, the timing controller in the electronic display 12, a separate processing module, or any combination thereof. Additionally, in some embodiments, the controller memory can be included in the local memory 20, the main memory storage device 22, a separate tangible, non-transitory computer-readable medium, or any combination thereof.

[0059] In some embodiments, memory 20 may include a source buffer for storing source image data. Thus, in such embodiments, processing pipeline 36 may obtain (e.g., retrieve) source image data from the source buffer for processing. In some embodiments, electronic device 10 may include multiple pipelines (e.g., processing pipeline 36, display pipeline, etc.) implemented to process image data. To facilitate communication, image data may be stored in memory 20 external to the pipeline. In such embodiments, a pipeline (e.g., processing pipeline 36) may include a direct memory access (DMA) block for reading (e.g., retrieving) and / or writing (e.g., storing) image data in memory 20.

[0060] After receiving from memory 20, the processing pipeline 36 may process the source image data via one or more image processing blocks, such as a scale and rotation block 42 or other processing blocks 44 (e.g., a dither block). In the depicted embodiment, the scale and rotation block 42 includes a scaler block 46 and may also include other modification blocks 48 (e.g., a rotation block, a flip block, a mirror block, etc.). As will be described in more detail below, the scaler block 46 may adjust the image data (e.g., via directional scaling and / or enhancement), for example, to facilitate reducing the likelihood of or correcting image artifacts typically associated with scaling. As an illustrative example, it may be desirable to increase the resolution of the image data to expand the viewing of the corresponding image or to accommodate the resolution of the electronic display 12. To achieve this, the scaler block 46 may employ noise statistics and / or SAD and DIFF statistics to analyze the content of the image data and scale the image data to a higher resolution while maintaining image clarity (e.g., sharpness). In some embodiments, the image data may also undergo enhancement.

[0061] For ease of explanation, Figure 7is a block diagram of a scaler block 46 that receives input image data 50 and outputs processed image data 52. The scaler block 46 may include a plurality of processing blocks 54 to perform directional scaling and / or enhancement. For example, the scaler block 46 may include a transform block 56, a noise statistics block 58, an angle detection block 60, a directional scaling block 62, an image enhancement block 64, and a vertical / horizontal scaling block 66.

[0062] The processing block 54 of the scaler block 46 can receive and / or process input image data 50 in a variety of color bases (e.g., red-green-blue (RGB), alpha-red-green-blue (ARGB), luminance-chrominance (YCC format such as YCbCr or YUV), etc.) and / or bit depths (e.g., 8-bit, 16-bit, 24-bit, 30-bit, 32-bit, 64-bit, and / or other suitable bit depths). Furthermore, high dynamic range (HDR) image data (e.g., HDR10, perceptual quantizer (PQ), etc.) can also be processed. However, in some embodiments, it may be desirable to utilize a channel representing luma values (e.g., the Y channel) to process or generate statistics from the input image data 50. A single luma channel can preserve the content of the image (e.g., edges, angles, lines, etc.) for pixel statistics collection and interpretation for directional scaling and enhancement. Based on the color basis of the input image data 50, the transform block 56 can generate luma pixel data representing a target white point, black point, or gray point for the input image data 50. This luminance pixel data may then be used by other processing blocks 54. For example, if the input image data 50 is in RGB format, the transform block 56 may apply weighting coefficients to each channel (i.e., the red channel, the green channel, and the blue channel) and combine the results to output a single channel of luminance pixel data. Additionally or alternatively, the processing blocks 54 may use the non-luminance pixel data to collect and interpret pixel statistics and for directional scaling and enhancement.

[0063] In one embodiment, the noise statistics block 58 may receive luma pixel data corresponding to the input image data 50. The noise statistics block 58 may then process the luma pixel data through one or more vertical and / or horizontal filters and quantify the qualification of the luma pixel data corresponding to each pixel. The qualified luma pixel data may be used to generate noise statistics from which the noise statistics block 58 may identify patterns in the image data content, such as for use in the image enhancement block 64. The image enhancement block 64 may take the scaled image data and / or the input image data 50 and enhance (e.g., sharpen) the image data using tone detection, comparison between the low-resolution input and the high-resolution input, and the noise statistics to generate enhanced image data.

[0064] Angle detection block 60 may also receive luminance pixel data corresponding to the input image data 50. The angle detection block 60 may analyze SAD statistics and DIFF statistics at multiple angles around a pixel of interest to identify angles corresponding to lines and / or edges within the content of the input image data 50. These angles may then be used in directional scaling block 62 to facilitate improved interpolation of new pixels generated when scaling to a higher resolution (e.g., doubling the size of the original image results in approximately four times the number of pixels). Additionally or alternatively, vertical / horizontal scaling block 66 may also scale the scaled image data to a higher or lower resolution to match the desired output resolution.

[0065] For ease of explanation, Figure 8 6 is a flowchart 68 depicting one embodiment of a process performed by scaler block 46. As described above, if necessary, scaler block 46 may transform input image data 50 into luma pixel data, for example, using transform block 56 (process block 70). The luma pixel data may be used, for example, to determine noise statistics via noise statistics block 58 (process block 72). The luma pixel data may also be used to determine SAD statistics and / or DIFF statistics (process block 74), which may then be used for angle detection, for example, using angle detection block 60 (process block 76). Scaler block 46 may also scale input image data 50 based at least in part on the detected angle, for example, via directional scaling block 62 (process block 78). Using input image data 50 or scaled image data, luma pixel data and / or chroma pixel data may be enhanced, for example, via image enhancement block 64, to generate enhanced image data (process block 80). Furthermore, if necessary, scaler block 46 may also perform vertical and / or horizontal scaling of the image data, for example, via vertical / horizontal scaling block 66 (process block 82).

[0066] As described above, the noise statistics block 58 may take the luminance pixel data 84 and generate noise statistics 86, such as Figure 9 As shown. Noise statistics 86, based at least in part on the content of the input image data 50, can allow noise to be distinguished from more interesting features of the image (e.g., features that are desired to be enhanced). In some embodiments, noisy aspects of the image are ignored when undergoing enhancement. To facilitate determining such noise, the noise statistics block 58 can include a pixel qualification sub-block 88, a vertical filter 90, and / or a horizontal filter 92.

[0067] The luma pixel data 84 may be processed in one or more vertical filters 90 and / or horizontal filters 92 in series, sequence, and / or parallel. Such filters 90 and 92 may include, for example, low-pass filters, band-pass filters, and / or high-pass filters capable of identifying and / or generating frequency content corresponding to different frequency bands of the image. The pixel qualification sub-block 88 may use the luma pixel data 84 and / or filtered luma pixel data to determine whether the pixel data for each individual pixel qualifies for use in the noise statistics 86. In some embodiments, the noise statistics block 58 may sample each pixel of the luma pixel data 84. However, in some embodiments, such as if the directional scaling block 62 is not enabled, the noise statistics block 58 may sample less than a complete set of luma pixel data 84 (e.g., one in every four pixels).

[0068] Qualified pixels are determined so that they meet one or more criteria. For example, in one embodiment, if a pixel falls into the valid region, the pixel is eligible for noise statistics 86. In some embodiments, the valid region can be set to group relevant parts of the image together while excluding irrelevant parts of the image. For example, the valid region can be set to exclude subtitles, constant color segments (e.g., letterboxes), etc. In addition, the valid region can be programmable and / or software-optimizable to increase the possibility of detecting various forms of noise from various content. For example, movie content may include artificial noise, such as film grain, that is intentionally added to each video frame. It may be desirable to use a specific valid region size to detect such artificial noise to increase or decrease enhancement. In some embodiments, the valid region may include the entire image.

[0069] Additionally or alternatively, other criteria may also apply, for example, the percentage of pixels in a window around the pixel of interest (e.g., a 1×1 (pixel itself) or a 3×3 pixel window) may contain luminance values within a specified range. Furthermore, a local activity metric (e.g., the sum of filtered or unfiltered luminance values of neighboring pixels) may also qualify a pixel for use in noise statistics 86 if the local activity metric is greater than a threshold.

[0070] Once qualified, the filtered and / or unfiltered luminance pixel data 84 can be used to form noise statistics 86 by determining, for example, a histogram, sum, maximum, minimum, mean, variance, and / or blockiness metric (e.g., a measure of corners and / or edges). Blockiness metrics can indicate artifacts resulting from, for example, block-based compression techniques. Such noise statistics 86 can represent global luminance values within a valid region and / or local values within a pixel window (e.g., a 5×5 pixel window). Noise statistics 86 can represent frequency signatures (e.g., frequency bands) of the image data that indicate blockiness, noise, or specific features that may be contained within the image (e.g., film grain, video capture noise, etc.). In some embodiments, features such as film grain are intentional within the image and, therefore, are desirable to be enhanced and scaled appropriately. Different image features may have different frequency signatures, and, therefore, these features can be determined based on analysis of programmed frequency bands. Distinguishing such features from noise allows for improved scaling and enhancement of desired image features and reduced enhancement of noise or noisy areas within the image.

[0071] Figure 10 Flowchart 94 depicts an embodiment of a process for determining noise statistics 86. Noise statistics block 58 may first receive luma pixel data 84, for example, from input image data 50 or as transformed by transform block 56 (process block 96). Noise statistics block 58 may then apply vertical and / or horizontal filters to the luma pixel data (process block 98). Noise statistics block 58 may also determine the eligibility of the luma pixel data (process block 100). Qualified luma pixel data may populate an update of noise statistics 86 (process block 102). For example, noise statistics 86 may be updated by generating a histogram, maximum, sum, mean, variance, blockiness metric, and / or other suitable metric of the qualified luma pixel data (process block 104). Noise statistics block 58 may also distinguish noise and frequency signatures (e.g., pre-programmed frequency bands) of desired content (e.g., film grain) from the content of the image (process block 106). Noise statistics 86 may then be output for use in image enhancement and / or other image processing techniques.

[0072] Similar to the noise statistics block 58, the angle detection block 60 may also take as input the luminance pixel data 84 on which statistics are determined. Based on such statistics, Figure 11As depicted, the angle detection block 60 may generate best pattern data 108. In one embodiment, the best pattern data 108 may include one or more angles corresponding to lines and edges of the image. In addition, the best pattern data 108 for each sampled pixel may include one or more angle weights corresponding to a confidence level of the angle and / or the similarity of the angle to those of neighboring pixels. The best pattern data 108 may facilitate improved directional scaling of the image data. The angle detection block 60 may include a SAD and DIFF statistics collection sub-block 110 with modifiers 112, a classifier 114, a high frequency and low angle detection sub-block 116, an angle consistency and difference setting sub-block 118, and a pattern and weight determination sub-block 120. The generation and analysis of the SAD and DIFF statistics, along with the assessment of confidence and consistency, may produce the best pattern data 108.

[0073] To generate SAD and DIFF statistics, the SAD and DIFF statistics collection sub-block 110 may analyze the luminance pixel data 84 in multiple directions around each pixel of interest. For example, Figure 12 A plurality of pixel groups 122 are shown for evaluating luma pixel data 84 at different angles. In some embodiments, the rectangular base pixel cluster 124 serves as a reference from which to determine the SAD and DIFF statistics for a pixel of interest. In some embodiments, the pixel of interest in the rectangular base pixel cluster is the top left pixel, however, other pixel positions may also be used. When compared to the rectangular base pixel cluster 124, the offset pixel clusters 126, 128, 130, 132, 134, 136, 138, 140, and 142 may yield information about how the luma pixel data 84 changes in different directions corresponding to the offset pixel clusters 126, 128, 130, 132, 134, 136, 138, 140, and 142. For example, the offset pixel clusters 126 and 128 may correspond to a 45-degree offset from the rectangular base pixel cluster 124. A 135-degree offset, orthogonal to the 45-degree offset, may be represented by the offset pixel clusters 130 and 132. In addition, vertical offset clusters 134 and 136 and horizontal offset clusters 140 and 142 may also be analyzed. In the event that a pixel cluster includes a pixel position that is not within the active area, the pixel value of the nearest pixel within the active area may be replaced. In one embodiment, the pixel values of pixel positions on the edge of the active area may be repeated horizontally and vertically to define the values of pixels outside the active area.

[0074] To represent other angles (e.g., angles with slopes other than 0, -1, 1, or infinity), diagonal basis pixel clusters 144, 146, 148, 150, 152, and 154 may be considered, as shown in FIG. Figure 13As shown. Like the rectangular basis pixel cluster 124, the diagonal basis pixel clusters 144, 146, 148, 150, 152, and 154 can be shifted by an offset and compared to obtain SAD and DIFF statistics corresponding to the corresponding angles. In some embodiments, some angles can be better represented by using a greater number of pixels in the pixel clusters. For example, when collecting SAD and DIFF statistics at a slope of 1 / 2, the diagonal basis pixel cluster 144 can be used, and the diagonal basis pixel cluster 152 can be used at a slope of 1 / 6. In this way, the diagonal basis pixel cluster 144 can utilize more pixels than the rectangular basis pixel cluster 124 and utilize fewer pixels than the diagonal basis pixel cluster 152.

[0075] The SAD and DIFF statistics collection sub-block 110 can calculate a metric for each desired angle using the sum of absolute differences (SAD) between the base and offset pixel groups 122. The angles shown by the pixel groupings 122 are shown by way of example and are therefore non-limiting. In some embodiments, the evaluation of the content of the image can be performed under various types of gradients (e.g., slopes, curves, angles, etc.). In addition to using SAD, difference (DIFF) statistics can also be collected. DIFF statistics can include metrics such as differences between consecutive pixels (e.g., in a line or curve), edge metrics for determining corners and / or edges within the image content, and / or other metrics.

[0076] In some embodiments, it may be desirable to reduce the bit depth of the input image data 50 or other data (e.g., luminance pixel data 84) to a smaller bit depth for ease of manipulation and / or analysis. This bit depth reduction can be achieved by, but is not limited to, shifting operations, clamping, clipping, and / or truncation. For example, before calculating the SAD and DIFF statistics, the bit depth of the analyzed data can be reduced by a shifting operation and then by clamping to reduce resource overhead (e.g., time, computational bandwidth, hardware capabilities, etc.). Additionally or alternatively, the bit depth reduction can be scalable based on a programmable parameter to set the desired bit depth reduction amount, which can vary depending on the specific implementation (e.g., whether for high-definition image processing or low-definition image processing). The bit reduction can produce a bit depth of any granularity, which may or may not be a multiple of 2. Furthermore, the bit reduction can also be used in other processing blocks 54 to reduce resource overhead.

[0077] The SAD and DIFF statistics collection sub-block 110 may also include a modifier 112 to normalize the angular statistics and account for the different numbers of pixels used at different angles. In some embodiments, the analysis of each angle and / or metric may be further adjusted by the modifier 112 based on the angle being examined. In some scenarios, lower angles (e.g., those with slopes less than 1 / 3 or 1 / 4 or greater than 2 or 3) may be prone to false positives when undergoing SAD and DIFF analysis. As such, the confidence level in the low-angle analysis may be lower than the confidence level in the horizontal or vertical directions, and therefore the low-angle analysis may be adjusted accordingly, for example, via the modifier 112. Based on the SAD and DIFF analysis, the classifier 114 may determine one or more optimal angles. The optimal angle may correspond to the angle that is closest to a uniform direction in the image content (e.g., a line, edge, etc.). In some embodiments, the classifier 114 generates the optimal angle and a second optimal angle for further consideration in the angle detection block 60.

[0078] In some embodiments, the horizontal and vertical directions can be processed separately from the other angles analyzed by the angle detection block 60. For example, depending on the interpolation method used during scaling, it may be desirable to interpolate diagonally positioned pixels at angles other than vertical or horizontal. On the other hand, it may also be desirable to use vertical and horizontal interpolation for pixels that are positioned directly perpendicular or horizontal to the original pixel. Thus, in some embodiments, the classifier 114 can output the best and second best angles among the non-vertical / horizontal angles, as well as the best vertical or horizontal angle.

[0079] After calculating one or more optimal angles, the angle detection block 60 may utilize the high frequency and low angle detection sub-block 116 to further evaluate the determined optimal angles. In some scenarios, the content of the image may have high frequency features (e.g., a checkerboard pattern) that may result in angle indications that do not accurately represent the image (e.g., false angles). The high frequency and low angle detection sub-block 116 may search for such high frequency features, for example, using horizontal and vertical DIFF statistics. In some embodiments, the optimal angles may be used to interpolate intermediate pixel values between the pixel values of the original pixels, and the high frequency and low angle detection sub-block 116 may check whether the approximate interpolation is consistent with neighboring pixels.

[0080] Additionally or alternatively, the high frequency and low angle detection sub-block 116 may utilize one or more conditions and / or parameters to determine the feasibility of the determined low angle. For example, differences between consecutive and / or consecutively considered pixel values (e.g., based on the luminance pixel data 84) may be considered. For a given set of pixels, these differences (e.g., positive differences, negative differences, or zero differences) may be considered together in a run (e.g., a run of positive differences, negative differences, or no differences in pixel values). The length of such a run may be calculated and used as a parameter of one or more low angle conditions to identify one or more low angle dilemmas. For example, if the length of a run of pixel value differences is within a configurable range (e.g., less than and / or greater than a threshold value set based on a specific implementation), the detected angle may be a false angle (e.g., a false edge detected due to noise), and the confidence level of the determined angle may be increased or decreased accordingly.

[0081] In some embodiments, identifying and calculating the conditions and / or parameters of the high frequency and low angle detection sub-block 116 while maintaining the data throughput of the angle detection block 60 may be expensive to implement in hardware. For example, an 8-bit or 16-bit implementation per pixel value may use eight or sixteen duplicate logic circuits, respectively, to determine the run length within a given time period. However, in some embodiments, a logic circuit design that combines forward and backward propagation can provide a single logic circuit that is scalable to multiple different implementations at minimal data path speed cost (e.g., less than 5%, 10%, or 20% per doubling of the bit depth of the luma pixel data 84) without changing or duplicating the logic circuits. In this way, the high frequency and low angle detection sub-block 116 can efficiently check the conditions and / or parameters to help identify the confidence level of the optimal angle.

[0082] In addition, if the best angle and / or the second best angle is a low angle relative to the horizontal and vertical (e.g., a slope less than 1 / 3 and greater than 3) and a high-frequency feature or a low-angle dilemma is detected, the confidence of the low angle may be lowered. In one embodiment, if the best angle is a low angle, the second best angle is not a low angle, and a high-frequency feature is detected, the second best angle may be output from the high-frequency and low-angle detection sub-block 116 as the new best angle, and the old best angle may become the new second best angle.

[0083] Furthermore, the angle detection block 60 may also include an angle consistency and difference setting sub-block 118. In some embodiments, it may be desirable to use (e.g., for interpolation or comparison) an orthogonal angle with the optimal angle. The angle consistency and difference setting sub-block 118 may determine an angle orthogonal to the optimal angle from previously analyzed angles. In this way, each angle analyzed in the SAD and DIFF statistics collection sub-block 110 may have an orthogonal or nearly orthogonal counterpart that has also been analyzed. Furthermore, in some embodiments, the optimal angle and the second optimal angle may be converted to angle metrics and compared to each other. If the difference between the optimal angle and the second optimal angle is less than a threshold, they may be considered consistent. Angle consistency may increase the confidence level of the optimal angle and / or decrease the confidence level if the angles are inconsistent. Furthermore, in some embodiments, the confidence level of the optimal angle may be compared to the confidence level of its orthogonal angle to further modify the confidence level. For example, if the confidence level of the existence of a line or edge in the image content in the orthogonal direction is nearly as high as the confidence level of the optimal angle, the confidence level of the optimal angle may be decreased.

[0084] The outputs of the SAD and DIFF statistics collection sub-block 110, the classifier 114, the high frequency and low angle detection sub-block 116, and / or the angle consistency and difference setting sub-block 118 may be fed into the pattern and weight determination sub-block 120. The pattern and weight determination sub-block 120 may determine the best pattern data 108 corresponding to the best angle, best horizontal / vertical angle, and / or orthogonal angle for each sub-block. Furthermore, the best pattern data 108 may include weights based at least in part on the confidence level of the angles. Furthermore, the weights assigned to the best angle and best horizontal / vertical angle at a particular pixel location may also be based at least in part on the best angles of neighboring pixel locations. For example, if most pixels surrounding a pixel of interest have the same best angle as the pixel of interest, the confidence level, and therefore the weight, of the best angle for the pixel of interest may be increased.

[0085] For ease of explanation, Figure 14Flowchart 156 depicts the operation of angle detection block 60 for a single pixel location. Angle detection block 60 may first determine the SAD and DIFF statistics at multiple angles from luma pixel data 84 (process block 158). The determined SAD and DIFF statistics may be normalized / modified, for example, based on individual angles (process block 160). Of the analyzed angles, one or more optimal angles may be determined, for example, by classifier 114 (process block 162). Using the optimal angles, angle detection block 60 may detect high-frequency and low-angle occurrences for possible undesirability (process block 164) and adjust the angle confidence accordingly. Furthermore, angle detection block 60 may determine angular consistency between a first optimal angle and a second optimal angle (process block 166) and determine angles orthogonal to the optimal angle (process block 168). As described above, vertical and horizontal angles may be processed separately from the rest, and thus, the optimal horizontal / vertical angles and corresponding orthogonal angles may also be included. Neighboring pixels may also be checked for consistency with the determined optimal angles (process block 170), for example, to update the angle confidence. Angle detection block 60 may then output optimal mode data 108 including the optimal angle and corresponding weight (process block 172 ), eg, for use in directional scaling block 62 .

[0086] When received by the directional scaling block 62, the best mode data 108 and the input image data 50 may be combined to generate scaled image data 174, as shown. Figure 15 In one embodiment, the directional scaling block 62 may include a midpoint interpolation sub-block 176 and an outside point interpolation sub-block 178. Although the above description describes the use of luma pixel data 84 for angle analysis, other color channels may also be used to collect statistics for angle detection and directional scaling. Furthermore, the best mode data 108 collected from a single channel may be used to scale multiple color channels. Thus, for each color channel, the same weights or their derivatives and angles used for interpolation may be used in the midpoint interpolation sub-block 176 and the outside point interpolation sub-block 178.

[0087] In some embodiments, pixel grid 180 may schematically represent the locations and relative positions of pixels, such as Figure 16As shown. Directional scaling block 62 may use input pixels 182 to interpolate midpoint pixel 184 and outer pixel 186. Furthermore, in some embodiments, midpoint pixel 184 is interpolated before outer pixel 186. Due to a lack of pixel data in the vertical or horizontal directions surrounding midpoint pixel 184, midpoint pixel 184 is interpolated diagonally using optimal angles, orthogonal angles, and / or weights from optimal pattern data 108. Using neighboring input pixels 182, midpoint interpolation sub-block 176 may determine a value for each color channel of each midpoint pixel 184. The interpolated value for each surrounding input pixel 182 is weighted at least in part based on the weights in optimal pattern data 108. Thus, the weights in the optimal pattern data may correspond to the weights in a weighted average of neighboring pixel values. In some embodiments, a temporary pixel value may be established by interpolating two or more input pixels 182. This temporary pixel value may then be used to interpolate midpoint pixel 184. Such a temporary pixel value may be used to better interpolate the value of midpoint pixel 184 at a specific angle. Furthermore, in some embodiments, a horizontal / vertical interpolation of the midpoint pixel 184 may be generated based at least in part on the optimal vertical / horizontal angle and blended with the diagonal interpolation to generate the value of the midpoint pixel 184 .

[0088] Once the midpoint pixel 184 has been determined, the outer point pixels 186 can be determined. Unlike the midpoint pixel 184, each outer point pixel 186 has input or vertically and horizontally determined pixel data around it. In this way, the optimal vertical / horizontal angle and weight can be used to interpolate the outer point pixels 186. For example, if the determined optimal vertical / horizontal angle is in the vertical direction, the outer point pixels 186 can be interpolated with a higher weight given to the pixels above and below the outer point pixel 186. It should be understood that a combination of vertical / horizontal optimal angles and diagonal optimal angles can also be used for midpoint pixel interpolation or outer point interpolation. In addition, in some embodiments, the outer point pixels 186 can be determined before the midpoint pixel 184.

[0089] In some embodiments, the directional scaling block 62 may scale at a fixed rate, such as by multiplying a dimension by 2, 4, or the like. To achieve higher or lower levels of resolution scaling, the directional scaling block 62 may be implemented multiple times (e.g., in cascade), and / or the vertical / horizontal scaling block 66 may be used to achieve non-multi-resolutions (e.g., resolutions of 1.2, 2.5, 3.75, or other multiples of the input resolution). The vertical / horizontal scaling block 66 may include a linear scaler, a polyphase scaler, and / or any suitable scaler to achieve the desired resolution. Furthermore, scaling may be implemented such that each dimension has a different scaling factor. Additionally or alternatively, the vertical / horizontal scaling block 66 may scale the input image data 50 in parallel with the directional scaling block 62. In this case, the output of the vertical / horizontal scaling block 66 and the output of the directional scaling block 62 may be mixed to generate scaled image data 174.

[0090] In further explanation, Figure 17 1 is a flow chart 188 depicting a simplified operation of directional scaling block 62. Directional scaling block 62 may first receive input image data 50 and best mode data 108 (process block 190). Directional scaling block 62 may also interpolate midpoint pixels 184 diagonally between input pixels 182 (process block 192) and interpolate outer point pixels 186 from input pixels 182 and midpoint pixels 184 (process block 194). Scaled image data 174 may then be output (process block 196).

[0091] In some embodiments, the scaled image data 174 may be sent to the image enhancement block 64. The image enhancement block 64 may also be used outside of the scaler block 46. In fact, in some embodiments, the image enhancement block 64 may enhance the input image data 50 without scaling it to a higher resolution. Figure 18 As depicted, the image enhancement block 64 may take as input the scaled image data 174, the input image data 50, or both, and the noise statistics, and output enhanced image data 198. If the scaled image data 174 is not available, the image enhancement block 64 may utilize the downsampling sub-block 200 to sample the input image data 50 (e.g., 1 in 4 pixels). This may also correspond to the sampling of the noise statistics block 58, where a subsample of the input image data 50 may be used if the directional scaling block 62 is disabled. The downsampled image data may be used as a low-resolution input for example-based refinement. If the scaled image data 174 is available, the input image data 50 may be used as the low-resolution input.

[0092] The image enhancement block 64 may also include a hue detection sub-block 202, a luminance processing sub-block 204, and a chrominance processing sub-block 206. The hue detection sub-block 202 may search the image content for a recognizable hue that matches a possible image representation (e.g., sky, grass, skin, etc.). In some embodiments, the hue detection sub-block 202 may combine multiple color channels to determine whether a recognizable hue is present. Furthermore, in some embodiments, the hue detection sub-block 202 may convert one or more color channels into a hue, saturation, value (HSV) format for analysis. A confidence level may also be assigned to each searched hue based at least in part on the likelihood that the detected hue is characteristic of the image representation. By including an improved assessment of the image content, the identification of hues within the image may result in improved enhancement of the image. For example, the luminance processing sub-block 204 and the chrominance processing sub-block 206 may use the hue data to adjust (e.g., with increased or decreased enhancement) the luminance and chrominance values of the input image data 50 or scaled image data 174 in the region where the hue is detected. In one embodiment, the effects on regions of various hues may be software programmable.

[0093] In one embodiment, the luminance processing sub-block 204 enhances (e.g., sharpens) the luminance channel of the input image data 50 or scaled image data 174 and includes luminance transition improvements 208 and example-based improvements 210. Luminance transition improvements 208 may include one or more horizontal or vertical filters (e.g., high-pass and / or low-pass) arranged with adaptive or programmable gain logic as peaking filters. The peaking filters may increase a programmable range of frequencies corresponding to content features of the image (e.g., cross-hatching, other high-frequency components). The increased frequency range may provide better frequency and / or spatial resolution for the luminance channel. Furthermore, luminance transition improvements 208 may include coring circuitry to minimize the amount of luminance enhancement in noisy regions of the input image data 50, for example, as determined by the noise statistics block 58. Furthermore, luminance transition improvements 208 may use an edge metric (e.g., determined from the SAD and DIFF statistics collection sub-block 110) within the coring circuitry to reduce overshoot and / or undershoot that may occur near edge transitions, for example, due to the increased frequency range.

[0094] Furthermore, the example-based improvement 210 may be run as part of the luminance processing sub-block 204 in parallel or in series with the luminance transition improvement 208. The example-based improvement 210 may take a low-resolution input 214 and compare a segment thereof (e.g., a 5×5 pixel segment) with a segment of a high-resolution input 216 (e.g., the input image data 50 or the scaled image data 174), as shown. Figure 19As shown. In some embodiments, the example-based improvement 210 may collect multiple (e.g., 25) segments (e.g., 5×5 pixel segments) of the low-resolution input 214 and compare each segment to a single segment of the high-resolution input 216. In addition, the low-resolution input 214 may be passed through a filter 218 (e.g., a low-pass filter) to generate a filtered low-resolution input 220. The high-resolution input 216, the low-resolution input 214, and / or the filtered low-resolution input 220 may be evaluated in a comparison and weight sub-block 222. For example, the comparison and weight sub-block 222 may utilize a sum of squared differences of the luma channel values or a squared difference approximation. In some embodiments, employing the sum of squared differences may be a resource (e.g., time, computational bandwidth) intensive process, and therefore it may be desirable to utilize a squared difference approximation instead.

[0095] A squared difference approximation can be implemented between each value of a segment of the low-resolution input 214 and each value of a segment of the high-resolution input 216. In one embodiment, a single squared difference can be estimated using a function that returns a bitwise value corresponding to the number of leading zeros in a segment of the high-resolution input 216, where the bitwise value corresponds to the difference between the value of the segment of the high-resolution input 216 and the corresponding value of the segment of the low-resolution input 214. The sum of the squared difference approximations can then at least partially represent an approximation of the sum of squared differences between the segments of the high-resolution input 216 and the segments of the low-resolution input 214. A sum of squared difference approximation can be implemented between each of a plurality (e.g., 25) segments (e.g., 5×5 pixel segments) of the low-resolution input 214 and a single segment of the high-resolution input 216 for use in the comparison and weight sub-block 222. Other operations such as clipping, multiplication by a programmable parameter, bit shifting, etc. can also be included in the calculation of the squared difference approximation or the sum of squared difference approximation.

[0096] Based on the similarities and differences between the high-resolution input 216, the low-resolution input 214, and / or the filtered low-resolution input 220, the comparison and weight sub-block 222 can determine weights from which a weighted average of the inputs is generated. For example, the comparison and weight sub-block 222 can apply a lookup table to the similarities and / or differences to generate weights for the weighted average. Based at least in part on the generated weights, the blending sub-block 224 can combine the inputs to generate improved luma data 226. Additionally, the luma processing sub-block 204 can combine the improved luma data 226 from the example-based refinement 210 with peak and coring refinements based on, for example, gradient statistics of the luma transition refinement 208.

[0097] The gradient statistics may indicate a linear change in pixel values in a particular direction (e.g., in the x-direction and / or the y-direction relative to the pixel grid 180). For example, a weighted average of the change in pixel values in the x-direction may be combined with a weighted average of the change in pixel values in the y-direction to determine how to blend the improved luminance data 226 from the example-based improvements 210 with the peak and coring improvements of the luminance transition improvements 208. The combination of the example-based improvements 210, which may result in improved identification and display of dominant gradients within an image, and the luminance transition improvements 208, which may improve perceived texture in the image, may allow for an enhanced (e.g., sharpened) luminance channel output.

[0098] Similar to luma transition improvements 208, chroma processing sub-block 206 may include chroma transition improvements 212, which include a peaking filter and coring circuitry. In some embodiments, chroma transition improvements 212 may be further enhanced based at least in part on luma transition improvements 208. In some scenarios, if the luma channel is enhanced without compensating for the chroma channels, the image may appear oversaturated or undersaturated. Thus, chroma transition improvements 212 may utilize luma channel changes due to luma processing sub-block 204 when determining changes from chroma transition improvements 212. Additionally or alternatively, chroma transition improvements 212 may be disabled if, for example, there is little or no luma channel enhancement. As output from image enhancement block 64, enhanced image data 198 (e.g., enhanced luma and chroma channels) may represent a sharper and more vibrant image.

[0099] For further explanation, Figure 20 2 is a flowchart 228 illustrating an exemplary process of the image enhancement block 64. The image enhancement block 64 may receive the input image data 50 (process block 230) and determine whether the scaled image data 174 is available (decision block 232). If the scaled image data 174 is not available, the input image data 50 may be downsampled to serve as a low-resolution input 214 for the example-based improvements 210 (process block 234). However, if the scaled image data 174 is available, the scaled image data 174 may be received (process block 236) and the input image data 50 may be used as a low-resolution input 214 for the example-based improvements 210 (process block 238). Thus, the example-based improvements 210 may be determined (process block 240). Furthermore, the image enhancement block 64 may determine a hue detection, for example, via the hue detection sub-block 202 (process block 242). The image enhancement block 64 may also determine a luminance transition improvement 208 (process block 244) and a luminance channel output (process block 246). Chroma transition improvements 212 may also be determined, for example, using the luma channel output (process block 248 ), and the chroma channel outputs may be determined (process block 250 ). The luma channel output and the chroma channel outputs together form enhanced image data 198 .

[0100] In some embodiments, the enhanced image data 198 may be scaled after enhancement. For example, the enhanced image data 198 may pass through the vertical / horizontal scaling block 66 after enhancement. Additionally or alternatively, the enhanced image data 198 may be scaled in the directional scaling block 62 before and / or after enhancement. Scaling and / or enhancement may be cascaded multiple times until the desired resolution is achieved.

[0101] It should be understood that the various components of the scaler block 46 (e.g., transform block 56, noise statistics block 58, angle detection block 60, directional scaling block 62, image enhancement block 64, vertical / horizontal scaling block 66) may be enabled, disabled, or employed together or separately, depending on the specific implementation. In addition, the order of use within the scaler block 46 may also be changed depending on the specific implementation (e.g., switched, repeated, run in parallel or in series, etc.). Thus, although the flowcharts referenced above are shown in a given order, in certain embodiments, the process / decision blocks may be reordered, changed, deleted, and / or occur simultaneously. Furthermore, the referenced flowcharts are provided as illustrative tools, and other decision blocks and process blocks may be added depending on the specific implementation.

[0102] The above specific embodiments have been shown by way of example, and it should be understood that these embodiments are susceptible to various modifications and alternative forms. It should also be understood that the claims are not intended to be limited to the particular forms disclosed, but are intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the present disclosure.

[0103] The technology described and claimed herein is cited and applied to specific examples of a tangible and practical nature that significantly advance the art and is therefore not abstract, intangible, or purely theoretical. Furthermore, if any claim appended to the end of this specification contains one or more elements designated as "means for [performing] [the function]..." or "a step for [performing] [the function]...", then those elements will be construed under 35 U.S.C. § 112(f). However, for any claim containing elements designated in any other manner, those elements will not be construed under 35 U.S.C. § 112(f).

Claims

1. An electronic device comprising an enhancement circuit, the enhancement circuit being configured to enhance high-resolution image data to improve the perceived quality of an image corresponding to the high-resolution image data, wherein the enhancement circuit comprises: a hue detection circuit configured to determine one or more hues within the image; as well as A channel processing circuit, wherein the channel processing circuit is configured to: applying a luma transition adjustment to a luma channel of the high-resolution image data based at least in part on the one or more hues, applying a chroma transition adjustment to a chroma channel of the high-resolution image data based at least in part on the one or more hues, or both; and The high-resolution image data is compared to low-resolution image data, and example-based adjustments are applied to the high-resolution image data based at least in part on differences between segments of the high-resolution image data and segments of the low-resolution image data. 2 . The electronic device of claim 1 , wherein the one or more hues include a hue representing at least one of skin, sky, and grass.

3. An electronic device according to claim 1, wherein the channel processing circuit includes a luminance processing circuit and a chrominance processing circuit, wherein the luminance processing circuit is configured to apply the luminance transition adjustment and the example-based adjustment to the luminance channel of the high-resolution image data, and wherein the chrominance processing circuit is configured to apply the chrominance transition adjustment to the chrominance channel of the high-resolution image data. 4 . The electronic device of claim 3 , wherein the brightness transition adjustment comprises an increase in a frequency range within the brightness channel of the high-resolution image data. 5 . The electronic device of claim 4 , wherein the boosting of the frequency range is based at least in part on an output of a peaking filter, wherein the frequency range corresponds to a high frequency pattern. The electronic device of claim 5 , wherein the high frequency pattern comprises cross-hatching. 7 . The electronic device of claim 4 , wherein the brightness processing circuit is configured to determine the example-based adjustment in parallel with the brightness transition adjustment.

8. The electronic device of claim 3, wherein the chroma processing circuit is configured to enhance the chroma channels based at least in part on the enhancement of the luma channel by the luma processing circuit. 9 . The electronic device of claim 1 , comprising a down-sampling circuit configured to generate the low-resolution image data by sampling a portion of the high-resolution image data. 10 . The electronic device of claim 9 , wherein the portion of the high-resolution image data includes one of every four pixel values of the high-resolution image data.

11. The electronic device of claim 1, wherein the high-resolution image data is generated by scaling the low-resolution image data to a higher resolution.

12. The electronic device of claim 1 , wherein the enhancement circuit is configured to: receiving a high-resolution luminance channel of the high-resolution image data and a low-resolution luminance channel of the low-resolution image data, each corresponding to the image, wherein comparing the high-resolution image data to the low-resolution image data comprises comparing the high-resolution luminance channel to the low-resolution luminance channel to generate the difference value; generating the example-based adjustment based at least in part on the difference; Generating a brightness transition adjustment based at least in part on: Improvement of the frequency range of the high-resolution luminance channel; as well as minimizing the correction of undershoot or overshoot at the edges of the image; and The high-resolution luma channel is enhanced based at least in part on a blend of the example-based adjustment and the luma transition adjustment.

13. A method comprising: receiving, via an enhancement circuit, a high-resolution luminance channel and a low-resolution luminance channel, each corresponding to an image; comparing the high-resolution luminance channel to the low-resolution luminance channel via the enhancement circuit to generate a difference value; generating, via the enhancement circuitry, an example-based adjustment to the high-resolution luma channel based at least in part on the difference; Generating, via the enhancement circuitry, a luminance transition adjustment to the high-resolution luminance channel based at least in part on: Improvement of the frequency range of the high-resolution luminance channel; as well as minimizing said correction of undershoot or overshoot at edges of said image; enhancing, via the enhancement circuitry, the high-resolution luma channel based at least in part on a blend of the example-based adjustment and the luma transition adjustment, thereby generating an enhanced luma channel; as well as The enhanced luminance channel is output.

14. The method of claim 13, wherein the difference value is determined based at least in part on a squared difference approximation, and wherein the example-based adjustment to the high-resolution luma channel is determined by applying a lookup table to the difference value.

15. The method of claim 13, comprising reducing, via the enhancement circuit, an amount of enhancement of a portion of the high-resolution luma channel based at least in part on determining that the portion of the high-resolution luma channel contains noise.

16. The method of claim 13, comprising receiving chroma channels via the enhancement circuit and enhancing the chroma channels based at least in part on the enhanced luma channel.

17. A system comprising: One or more processors, the processors configured to: receiving high-resolution image data corresponding to an image; determining one or more tones within the image; generating a brightness transition adjustment to the high-resolution image data based at least in part on the one or more hues; comparing the high-resolution image data with the low-resolution image data; generating an example-based adjustment to the high-resolution image data based at least in part on a difference between a segment of the high-resolution image data and a segment of the low-resolution image data; as well as applying the luminance transition adjustment and the example-based adjustment to a luminance channel of the high-resolution image data to generate enhanced image data; as well as A controller is configured to retrieve the high-resolution image data from a memory for the processor and output the enhanced image data to an electronic display or the memory.

18. The system of claim 17, wherein the processor is configured to generate the low-resolution image data by downsampling the high-resolution image data.

19. The system of claim 17, wherein the processor is configured to limit enhancement in regions of the image based at least in part on noise statistics.

20. The system of claim 17, wherein the processor is configured to compare the high-resolution image data with the low-resolution image data via an improved circuit based on example.

21. The system of claim 17, comprising a display driver configured to drive the enhanced image data to an electronic display.

Images