Context-aware image processing

Abstract

A method and apparatus for image processing. An image scaling circuit is configured to receive image data for a plurality of pixels corresponding to a first image. The image data includes color information and an opacity value for each of the plurality of pixels. The image scaling circuit updates the image data by selectively altering the color information for one or more of the pixels based at least in part on the opacity value. In updating the image data, the image scaling circuit can determine context information about the first image based at least in part on the opacity value. The image scaling circuit also generates an interpolated image based on the updated image data for the plurality of pixels. For example, the interpolated image can be a scaled version of the first image.

Description

Technical Field

[0001] This embodiment generally involves image processing. Background Technology

[0002] Image processing enables captured images to be displayed on a monitor, allowing the original image to be reproduced as accurately as possible within the capabilities (or limitations) of a given display technology. For example, a high-definition (HD) display device with a horizontal resolution of 2,000 pixels may not be able to reproduce a full-resolution image captured in ultra-high-definition (UHD) format (e.g., with a horizontal resolution of 4,000 pixels). Therefore, image processing can reduce the number of pixels in the original image, making it possible to display the image on an HD display. The process of converting an image from its native resolution to a higher or lower resolution is commonly referred to as image scaling. Summary of the Invention

[0003] This summary is provided to present a simplified version of the conceptual options, which will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] Methods and apparatus for image processing are disclosed. One innovative aspect of the subject matter of this disclosure can be implemented in the image processing method. In some embodiments, the method may include the steps of: receiving image data for a plurality of pixels corresponding to a first image, wherein the image data includes color information and transparency values ​​for each of the plurality of pixels; updating the image data by selectively changing the color information for one or more pixels, at least in part based on the transparency values; and generating an interpolated image based on the updated image data. For example, the interpolated image may be a scaled version of the first image. When updating the image data, the method may further include a step of determining contextual information about the first image, at least in part based on the transparency values.

[0005] Another innovative aspect of this disclosed subject matter can be implemented in an image processing system. In some embodiments, the image processing system may include an encoding circuit and a scaling circuit. The encoding circuit receives first image data from a first image source and also receives second image data from a second image source. The encoding circuit is configured to generate third image data based on the first and second image data, wherein the third image data includes color information and transparency values ​​for each of a plurality of pixels corresponding to the third image. The scaling circuit is configured to update the third image data by selectively changing the color information for one or more pixels, at least in part based on the transparency values. The scaling circuit also generates an interpolated image based on the updated third image data. Attached Figure Description

[0006] This embodiment is shown by way of example and is not intended to be limited to the figures in the accompanying drawings.

[0007] Figure 1 A block diagram of an example video post-processing (VPP) pipeline configured to transfer images from different image capture devices to a display device is shown.

[0008] Figure 2 An example image processing system that can implement this embodiment is shown.

[0009] Figure 3A and 3B Example portions of the images before and after scaling are shown.

[0010] Figure 4 A block diagram of an image processing system according to some embodiments is shown.

[0011] Figure 5 An example section of a scaled image using context-aware interpolation is shown.

[0012] Figure 6 A block diagram of an image scaling circuit according to some embodiments is shown.

[0013] Figure 7 A block diagram of a context-aware pixel adjustment circuit according to some embodiments is shown.

[0014] Figures 8A-8C An example set of pixels from which interpolated pixels can be derived is shown.

[0015] Figure 9 Another block diagram of an image scaling circuit according to some embodiments is shown.

[0016] Figure 10 This is an illustrative flowchart describing example image processing operations according to some embodiments.

[0017] Figure 11 This is an illustrative flowchart describing example context-aware pixel adjustment operations according to some embodiments. Detailed Implementation

[0018] In the following description, numerous specific details, such as examples of particular components, circuits, and processes, are set forth to provide a thorough understanding of this disclosure. As used herein, the term "coupling" means a direct connection to or a connection via one or more intermediate components or circuits. Furthermore, in the following description, specific terminology is set forth for purposes of explanation to provide a thorough understanding of aspects of this disclosure. However, it will be apparent to those skilled in the art that these specific details may not be necessary for practical example embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring this disclosure. Some portions of the following detailed description are presented according to other notational representations of processes, logic blocks, handling, and operations on data bits within computer memory. Interconnections between circuit elements or software blocks may be shown as buses or single signal lines. Each of a bus may alternatively be a single signal line, and each of a single signal line may alternatively be a bus, and a single line or bus may represent any one or more of a multitude of physical or logical mechanisms for communication between components.

[0019] Unless otherwise stated, it will be apparent from the following discussion that the use of terms such as “access,” “receive,” “send,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” and “derive” throughout this application refers to the actions and processes of a computer system or similar electronic computing device that manipulate and convert data represented as physical (electronic) quantities in the registers and memories of the computer system into other data represented as physical quantities in the computer system’s memory or registers or other such information storage, transmission, or display devices.

[0020] Unless specifically described as implemented in a particular manner, the techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Any feature described as a module or component may also be implemented together in an integrated logic device or separately as a discrete but interoperable logic device. If implemented in software, the techniques may be implemented at least in part by a non-transitory computer-readable storage medium comprising instructions that, when executed, perform one or more of the methods described above. The non-transitory computer-readable storage medium may form part of a computer program product, which may include packaging material.

[0021] Non-transitory processor-readable storage media may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and other known storage media. Additionally or alternatively, the technique may be implemented at least in part through a processor-readable communication medium that carries or transmits code in the form of instructions or data structures, and which can be accessed, read, and / or executed by a computer or other processor.

[0022] The various illustrative logic blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein can be executed by one or more processors. As used herein, the term "processor" can refer to any general-purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0023] This disclosure relates to systems and methods for context-aware image scaling. Image scaling is commonly used to convert digital images from their native resolution to a resolution suitable for a display device. The resulting image is a scaled (e.g., up-scaled or down-scaled) version of the original image. Because the scaled image will have fewer or more pixels than the original image, the pixel values ​​of the scaled image must be interpolated from the pixel values ​​of the original image. Interpolation often results in visual artifacts (e.g., blurring, fading, ringing, etc.) around the edges of objects in the scaled image. Context-aware interpolation (CAI) is a technique for reducing such visual artifacts, for example, by first determining the context of objects in the image and then using the context information to fine-tune the interpolation at the edges or boundaries of the objects. This embodiment provides a fast and inexpensive method for CAI that can be performed without a context detection step. In some embodiments, the image scaler may utilize pixel transparency information for the received image when determining how to interpolate pixel values.

[0024] Figure 1 A block diagram of an example video post-processing (VPP) pipeline 100 is shown, configured to pass images from different image capture devices to a display device. The VPP pipeline 100 includes a direct media access (DMA) controller 110, a main video channel 120, a sub-video channel 130, a graphics channel 140, and an overlay module 150. The VPP pipeline 100 can receive one or more incoming video signals from image capture devices such as cameras or video recorders, and process the received video signals for presentation on the display device.

[0025] DMA 110 can receive video input data 101 from various sources (e.g., image capture devices) and redistribute the video input data 101 to one or more of channels 120-140. For example, if the video input data 101 corresponds to a primary video feed (e.g., from a first source device), DMA 110 can forward the video input data 101 to the primary video channel 120. If the video input data 101 corresponds to a secondary video feed (e.g., from a second source device), DMA 110 can forward the video input data 101 to the sub-video channel 130. If the video input data 101 corresponds to graphics (e.g., from a third source device), DMA 110 can forward the video input data 101 to the graphics channel 140.

[0026] The main video channel 120 processes the video input data 101 to generate primary video data 102 for display on a corresponding display device. The primary video data 102 may correspond to a main video feed that is to be prominently displayed on the display device (e.g., by occupying most (if not all) of the display area). Therefore, the main video channel 120 may perform the maximum amount of post-processing on the video input data 101 (e.g., more post-processing than the sub-video channel 130 and graphics channel 140) to ensure that the primary video data 102 can be reproduced as accurately as possible with minimal noise and / or artifacts.

[0027] Sub-video channel 130 processes video input data 101 to generate secondary video data 103 for display on a corresponding display device. Secondary video data 103 may correspond to a secondary video feed that is presented simultaneously with the primary video feed in a relatively small display area of ​​the display device (e.g., in picture-in-picture or "PIP" format). Because the secondary video feed can occupy a much smaller display area than the primary video feed, sub-video channel 130 can perform less post-processing than primary video channel 120 (e.g., but more post-processing than graphics channel 140) when generating secondary video data 103.

[0028] Graphics channel 140 processes video input data 101 to generate graphics data 104 for display on a corresponding display device. Graphics data 104 may correspond to one or more graphics (e.g., as a HUD or overlay) presented simultaneously with the primary and / or secondary video feeds in a portion of the display device. Because the graphics may not contain detailed image or video content, graphics channel 140 may perform minimal post-processing when generating graphics data 104 (e.g., less than the post-processing of primary video channel 120 and sub-video channel 130).

[0029] The overlay module 150 can combine primary video data 102 with at least one of secondary video data 103 and / or graphics data 104 to produce video output data 105 corresponding to the combined video feed optimized for display on a display device. For example, each frame in the combined video feed may include a single frame of the primary video feed and a single frame of graphics and / or graphics displayed together with the frame of the primary video feed. In some implementations, the overlay module 150 may present the secondary video data 103 and / or graphics data 104 as an overlay map covering at least a portion of the primary video feed 102. Therefore, when the display device presents the video output data 105, at least some of the pixels will display a portion of the primary video feed, and at least some of the pixels will display the secondary video feed and / or graphics overlay.

[0030] Figure 2 An example image processing system 200 in which this embodiment can be implemented is shown. System 200 includes an image down-scaler 210, an image up-scaler 220, and an image mixer 230. Image processing system 200 can be... Figure 1 An embodiment of at least a portion of the overlay module 150. More specifically, the image processing system 200 may be configured to combine multiple images 202 and 204 into a single mixed image 206 for display on a display device. In some aspects, one or more of images 202, 204 and / or 206 may correspond to frames of video to be played back on the display device.

[0031] Image downscalorer 210 is configured to receive first image 202 and generate a corresponding downscaled (DS) image 203. Figure 2 In some embodiments, the first image 202 may correspond to a secondary video feed to be presented simultaneously with the primary video feed in a relatively small display area (such as a PIP window) of the display device. However, the resolution of the PIP window may be significantly lower than the native resolution of the first image 202. Therefore, the first image 202 may need to be resized (e.g., down-scaled) to fit the PIP window. The down-scaled image 203 may include fewer pixels than the first image 202. In some embodiments, the image down-scaler 210 may interpolate pixel values ​​for the down-scaled image 203 from the pixel values ​​of the first image 202. Examples of suitable down-scale interpolation techniques may include, but are not limited to, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc.

[0032] Image upscaler 220 is configured to receive second image 204 and generate a corresponding upscaled (US) image 205. Figure 2In some embodiments, the second image 204 may correspond to a primary video feed that is to be prominently displayed on a display device (e.g., by occupying most, if not all, of the display area). However, the overall resolution of the display device may be significantly larger than the native resolution of the second image 204. Therefore, the second image 204 may need to be resized (e.g., upscaled) to the fill resolution of the display device. The upscaled image 205 may include more pixels than the second image 204. In some embodiments, the image scaler 220 may interpolate pixel values ​​for the upscaled image 205 from the pixel values ​​of the second image 204. Examples of suitable upscaled interpolation techniques may include, but are not limited to, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc.

[0033] Image mixer 230 combines (e.g., blends) scaled images 203 and 205 to generate a blended image 206. Figure 2 In this embodiment, the up-scaled image 205 is a prominent feature of the blended image 206. In other words, the vast majority of pixels in the blended image 206 are copied from the up-scaled image 205. Conversely, the down-scaled image 203 is presented as an overlay in the upper-right portion of the blended image 206 (e.g., in a PIP window). Therefore, only a small subset of pixels in the blended image 206 are copied from the down-scaled image 203. When generating the blended image 206, the image mixer 230 can replace or substitute a small subset of pixels in the up-scaled image 205 (e.g., in a PIP window) with the pixel values ​​of the down-scaled image 203.

[0034] The second image 204 includes a framed area 201 corresponding to the PIP window in the blended image 206. Figure 2 In one embodiment, the framed area 201 is depicted as a black rectangle with different colored edges. The framed area 201 may be generated by a graphics generator rather than an image capture device and is integrated with the second image 204 before scaling. (See reference for example...) Figure 1 The graphic data 104 may include pixel values ​​for the framed area 201, and the primary video data 102 may include pixel values ​​for the remaining portion of the image 204. Before combining the primary video data 102 with the secondary video data 103, the overlay module 150 may first merge the graphic data 104 with the primary video data 102 to produce a second image 204 with the framed area 201.

[0035] After upscaling, the black rectangle inside the bounding region 201 (e.g., the up-scaled image 205) can have substantially the same resolution as the down-scaled image 203. Therefore, the image mixer 230 can replace the black pixels within the bounding region 201 with the corresponding pixel values ​​of the scaled image 203. In some implementations, for example, the edges surrounding the bounding region 201 can be maintained in the blended image 206 to depict the secondary video feed from the primary video feed. However, without context awareness, each pixel value in the up-scaled image 205 will be directly interpolated from a predetermined number (N) of pixel values ​​in the second image 204. This can result in visual artifacts (e.g., blurring, fading, ringing, etc.) around the boundaries or edges.

[0036] Figure 3A An example portion of image 310 before scaling is shown. Image portion 310 includes an edge 312 (A) that delineates or separates a first region of image 312 (B) from a second region of image 314. (See reference, for example...) Figure 2 Image portion 310 may be a close-up view of portion 207 of the second image 204 (e.g., the area outlined by the dashed circle). More specifically, edge 312(A) may correspond to the area surrounding... Figure 2 The edge of the framed area 201. Therefore, the first area of ​​image 312(B) can correspond to the black rectangle of the framed area 201, and the second area of ​​image 314 can correspond to the underlying image 204. Figure 3A As shown, edge 312(A) has sharp, clear edges 311 and 313, wherein edge 312(A) intersects with the second region of image 314.

[0037] Figure 3B An example portion of the scaled image 320 is shown. Image portion 320 includes an edge 322 (A) that delineates or separates a first region of image 322 (B) from a second region of image 324. Image portion 320 can be [the following text is missing here, likely due to an incomplete sentence or a formatting error] when upscaling is performed without context awareness. Figure 3A The image portion 310 shown is an upscaled version. More specifically, after upscaling, edge 322(A) can correspond to... Figure 3A Edge 312(A). In contrast to edge 312(A), edge 322(A) has blurred edges 321 and 323, where edge 322(A) intersects with the second region of image 324. Before scaling, edges 321 and 323 also exhibit phantom colors not found in any of the surrounding pixels of the original image 310. This is known as the "ringing" effect.

[0038] Artifacts (e.g., blurring and ringing) along edges 321 and 323 may be caused by pixel interpolation during the upscaling process. Because the upscaled image 320 includes a greater number of pixels than the original image 310, many (if not all) of the pixel values ​​for the upscaled image 320 (e.g., “upscaled pixels”) must be created or generated by an image upscaler (such as image upscaler 220). More specifically, when generating upscaled pixels, the image upscaler may approximate pixel values ​​based on neighboring pixels in the original image 310 (e.g., “original pixels”). For example, the image upscaler may determine the pixel values ​​for the upscaled pixels based on a weighted average of a number (N) of the pixel values ​​in the neighboring original pixels. At the edges or margins of objects in the upscaled image 320, each upscaled pixel value is at least partially derived from the pixel values ​​of adjacent objects and / or features. This causes edges 321 and 323 to exhibit blurring and / or ringing effects.

[0039] Context-aware interpolation (CAI) is a technique that reduces visual artifacts by first determining the context of objects in an image and then using the contextual information to fine-tune interpolation at the edges or boundaries of the objects. Example CAI techniques include temporal interpolation and spatial interpolation. Temporal interpolation involves detecting motion of objects across multiple image or video frames and determining object boundaries in each image based on the detected motion. Spatial interpolation involves detecting edges of objects in each image or video frame and determining object boundaries based on the detected edges. In contrast to temporal and spatial interpolation techniques, this embodiment can perform CAI without the additional step of detecting the context of the objects before interpolation can be performed. Because no additional processing is required to derive such contextual information, the CAI techniques disclosed herein are likely to be cheaper and simpler to implement.

[0040] This disclosure recognizes that some contextual information can be included in the raw image data. For example, raw image data for a given pixel may include color information (e.g., red, green, and blue component values) and a transparency value (α). The transparency value may be an 8-bit value specifying the transparency (or opacity) of a given pixel. While minute differences in transparency values ​​(such as between 254 and 255) may be substantially indistinguishable to the human eye, any difference in transparency values ​​can be readily identified by image processing hardware. Therefore, in some embodiments, image scaling circuitry (such as image downscaler 210 and / or image upscaler 220) can utilize the transparency values ​​included in the received image data to perform context-aware interpolation. More specifically, the image scaling circuitry can extract contextual information from the raw image data without further processing or analysis. Among other advantages, this embodiment provides a low-cost, low-complexity CAI solution that can be used to reduce artifacts in image scaling.

[0041] Figure 4 A block diagram of an image processing system 400 according to some embodiments is shown. System 400 includes an image encoder 410 and an image scaler 420. The image processing system 400 may be... Figure 1 An embodiment of at least a portion of the overlay module 150. More specifically, the image processing system 400 can be configured to generate scaled images for display on a display device. In some aspects, the scaled images can be generated by combining image data from multiple image sources.

[0042] Image encoder 410 is configured to receive image data 402 and 404 from multiple sources and generate encoded image data 406 by combining the received image data 402 and 404. In some aspects, the first image data 402 may be received from video channel 401, while the second image data 404 may be received from graphics channel 403. (See reference, for example...) Figure 1 Video channel 401 may correspond to main video channel 120 or sub-video channel 130, while graphics channel 403 may correspond to graphics channel 140. Therefore, second image data 404 may correspond to the graphics to be embedded in first image data 402. (See, for example, reference...) Figure 2 The encoded image data 406 may include pixel values ​​for the second image 204. Furthermore, the second image data 404 may include pixel values ​​for the framed area 201 (e.g., a PIP window), and the first image data 402 may include pixel values ​​for the remaining pixels of the second image 204.

[0043] Image encoder 410 can generate encoded image data 406 by replacing or substituting a subset of pixel values ​​in first image data 402 with pixel values ​​from second image data 404. For example... Figure 4 As shown, image encoder 410 receives first image data 402 and second image data 404 from different sources. Aspects of this disclosure recognize that different image sources can provide context for image data 402 and 404. For example, because second image data 404 is received via graphics channel 403, image encoder 410 can recognize that second image data 404 describes a permanent object or fixture in an encoded image (e.g., second image 204). In some embodiments, image encoder 410 can encode contextual information into encoded image data 406. For example, when generating encoded image data 406, image encoder 410 can encode pixel values ​​derived from first image data 402 differently than pixel values ​​derived from second image data 404.

[0044] In some embodiments, contextual information can be encoded using transparency values ​​associated with each pixel. More specifically, the transparency value associated with the first image data 402 can differ from the transparency value associated with the second image data 404 by at least a threshold amount. For example, assuming the encoded image data 406 is to be presented as an opaque image, the pixel value derived from the first image data 402 can have a transparency value of 255, while the pixel value derived from the second image data 404 can have a transparency value of 254. This disclosure recognizes that while such a small difference in transparency values ​​may be substantially indistinguishable to the human eye, it can easily distinguish the context of the first image data 402 from that of the second image data 404 for an image processor.

[0045] In some aspects, the image encoder 410 may include a context encoding module 412 to generate context information for the encoded image data 406. For example, based on different image sources from which first image data 402 and second image data 404 are received, the context encoding module 412 may encode the first image data 402 differently from the second image data 404. In some embodiments, the context encoding module 412 may adjust or modify the transparency values ​​for the first image data 402 and / or the second image data 404 to ensure that the transparency value for the first image data 402 differs from the transparency value for the second image data 404 by at least a threshold amount. For example, if the first image data 402 and the second image data 404 are received with a transparency value of 255, the context encoding module 412 may reduce the transparency value for the second image data 404 (e.g., reduce it to 254 or lower) when encoding pixels of the second image data 404 in the encoded image data 406.

[0046] In some other respects, contextual information can be generated by the image source itself. For example, the second image data 404 can be generated locally by a graphics generator (not shown for simplicity) residing on an image processing platform. To provide context to the second image data 404, the graphics generator can generate the second image data 404 in a manner different from how the generator would otherwise generate such image data to achieve the desired output. In some embodiments, the graphics generator can select a transparency value for the second image data 404 that differs sufficiently from the transparency value for the first image data 402 by at least a threshold amount. For example, if the second image data 402 is to be rendered as an opaque image or graphic on a display device, the graphics generator can use a transparency value (e.g., 254 or lower) that is slightly lower than the transparency value (e.g., 255) that would otherwise be used to achieve opacity when generating the second image data 404.

[0047] Image scaler 420 receives encoded image data 406 from image encoder 410 and generates scaled image data 408. Image scaler 420 can be... Figure 2 This is an embodiment of an image down scaler 210 or an image up scaler 220. Therefore, image scaler 420 can scale encoded image data 406 to different (higher or lower) resolutions, for example, to match the resolution of a display device. More specifically, image scaler 420 can use context-aware interpolation techniques to scale encoded image data 406. In some embodiments, image scaler 420 may include a context extraction module 422 to extract contextual information embedded in encoded image data 406, rather than determining such contextual information through additional processing and / or analysis. Image scaler 420 can then use the contextual information to interpolate pixel values ​​for scaled image data 408, for example, to produce smooth and / or sharp edges at the boundaries of objects in the scaled image.

[0048] In some respects, the context extraction module 422 can determine the context of one or more objects in the encoded image data 406 based at least in part on the transparency value for each pixel of the encoded image data 406. As described above, the transparency value for a pixel derived from the first image data 402 can differ from the transparency value for a pixel derived from the second image data 404 by at least a threshold amount. The difference in transparency values ​​can be interpreted by the context extraction module 422 as contextual information. More specifically, the context extraction module 422 can use the difference in transparency values ​​to identify object boundaries in the encoded image data 406. The image scaler 420 can then fine-tune the pixel interpolation at the object boundaries to prevent cross-contamination of pixel data from either side of the object boundaries.

[0049] By, for example, reference Figure 2 The context extraction module 422 can detect the boundary of the bounding region 210 within the second image 204 based on the transparency value of each pixel in the second image 204. For example, the transparency value for pixels within the bounding region 210 can differ from the transparency value for the remaining pixels in the second image 204 by at least a threshold amount. When the second image 204 is upscaled (e.g., to produce an upscaled image 205), the image scaler 420 can ensure that the pixel values ​​along the boundary or edge of the PIP window in the upscaled image 205 are interpolated entirely from the pixel values ​​within or outside the bounding region 210 of the original image 204 (but not both). This can reduce artifacts that would otherwise be created around the edges of the PIP window in the upscaled image 205 (e.g., in the absence of context awareness).

[0050] Figure 5An example portion of a scaled image 500 using context-aware interpolation is shown. Image portion 500 includes an edge 512 (A) that delineates or separates a first region of image 512 (B) from a second region of image 514. (See reference, for example...) Figure 3A Image portion 500 can be an embodiment of image portion 310 after being upscaled using CAI. For example, Figure 4 The image scaler 420 can scale the image portion 310 to the image portion 500. Therefore, Figure 5 The edge 512(A), the first region of image 512(B), and the second region of image 514 can respectively correspond to Figure 3A The edge 312(A), the first region of image 312(B), and the second region of image 314. For example... Figure 5 As shown, edge 512(A) has sharp, clear edges 511 and 513, where edge 512(A) intersects with the second region of image 514. Figure 3B Conversely, the edges 511 and 513 of image portion 500 do not exhibit any of the artifacts shown in the edges 321 and 323 of image portion 320.

[0051] This disclosure also recognizes that the encoded image data 406 can be used, in addition to or in lieu of image scaling, to provide context-awareness to other image processing operations. For example, image processing can also be used to reduce the color, brightness, and / or contrast of a high dynamic range (HDR) image to be presented on a standard dynamic range (SDR) display. Given the limitations of SDR displays, contextual information about graphics and / or objects in an HDR image can help determine how to accurately reproduce the image on an SDR display. For example, the image graphics may have HDR and / or SDR display characteristics different from other components of the image. Therefore, in some embodiments, transparency values ​​in the encoded image data 406 can be used as contextual information when converting an image from the HDR domain to the SDR domain, and vice versa.

[0052] Figure 6 A block diagram of an image scaling circuit 600 according to some embodiments is shown. The image scaling circuit 600 includes a pixel adjustment module 610 and a pixel interpolation module 620. The image scaling circuit 600 may be... Figure 4 One embodiment of the image scaler 420. More specifically, the image scaling circuit 600 may be configured to convert received image data 602 into interpolated image data 606, wherein the interpolated image data 606 represents a scaled version of the received image data 602.

[0053] Pixel adjustment module 610 is configured to receive image data 602 and generate updated image data 604 by selectively changing the pixel values ​​of one or more pixels of the received image data 602. In some embodiments, pixel adjustment module 610 may selectively change the color information of one or more pixels of the received image data 602 based at least in part on transparency values ​​for pixels. For example, transparency values ​​can provide context for one or more objects in the received image data 602. (See also: Regarding...) Figure 4 and 5 As described above, the edges of an object can be identified by changes in the transparency values ​​between pixels associated with the object and pixels outside the object (e.g., exceeding a threshold amount). To prevent or reduce artifacts along the edges of objects in the scaled image, only color information associated with the object in the original image can be used to interpolate the color information of the object in the scaled image.

[0054] In some embodiments, the pixel adjustment module 610 may change the color information for one or more pixels adjacent to the edge of an object for interpolation purposes. For example, if the pixel values ​​of the scaled image are interpolated from the number of pixels (n) within the boundary of the object in the original image and the number of pixels (m) outside the boundary of the object, the pixel adjustment module 610 may change the color information for m pixels in the updated image data 604. In some aspects, the pixel adjustment module 610 may change the color information for each of the m pixels to match the color information for one or more of the n pixels located within the boundary of the object. For example, the pixel adjustment module 610 may change the color information for each of the m pixels to the color information of the pixel in the original image whose position is closest to the corresponding pixel in the scaled image.

[0055] Pixel interpolation module 620 is configured to generate interpolated image data 606 based on updated image data 604. The interpolated image data 606 may include pixel values ​​for one or more pixels of the scaled image. More specifically, pixel interpolation module 620 may interpolate the color information for each pixel of the interpolated image data 606 from a number (N) of color information from the pixels of the updated image data 604 (e.g., N-tap interpolation). Suitable pixel interpolation techniques may include, but are not limited to, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc. Therefore, the color of each pixel of the interpolated image data 606 may depend on a weighted average of the colors of each of the N pixels of the updated image data 604.

[0056] Figure 7A block diagram of a context-aware pixel adjustment circuit 700 according to some embodiments is shown. The pixel adjustment circuit 700 includes an interpolated pixel position (IPP) detection module 710, a principal pixel detection module 720, a context comparison module 730, and a pixel replacement module 740. The context-aware pixel adjustment circuit 700 may be... Figure 6 This is one embodiment of the pixel adjustment circuit 610. Therefore, the pixel adjustment circuit 700 can generate updated pixel data 706 by selectively changing the pixel values ​​of one or more pixels of the received pixel data 703, at least partially based on context information. The updated pixel data 706 can be used to generate interpolated pixels for a scaled image. More specifically, the interpolated pixels can be derived (e.g., interpolated) from a number (N) of pixels in the original image.

[0057] The IPP detection module 710 is configured to determine the interpolated pixel position (IP_Pos) 712 based at least in part on scaling information 701. The interpolated pixel position 712 may correspond to the position of the interpolated pixel in the scaled image. For example, the scaling information 701 may include the scaling ratio to be performed on the original image and the initial phase (e.g., to produce the scaled image). This can be achieved by, for example, referring to... Figure 6 Alternatively, the pixel interpolation module 620 can use the scaling information 701 to generate the interpolated image data 606. Therefore, in some embodiments, the IPP detection module 710 can use the scaling information 701 to predict or predetermine the positions of the interpolated pixels to be generated by the pixel interpolation module 620.

[0058] The principal pixel detection module 720 is configured to select a principal pixel (P_Pixel) 722 among the N original pixels, at least in part based on the interpolated pixel position 712 for each of the N original pixels and position information 702. The position information 701 can indicate the relative positions of the N pixels from the original image. The principal pixel 722 can be the pixel in the original image that is positionally closest to the interpolated pixel position 712. For example, Figure 8A An example set of pixels P0-P7 is shown, from which the interpolated pixels can be derived. The position of the interpolated pixel 801 is depicted (in a dashed gray circle). Figure 8A In this embodiment, the interpolated pixel position 801 is located between pixels P3 and P4. However, the interpolated pixel position 801 is closer to the position of the black pixel P3 than the white pixel P4. Therefore, pixel P3 can be selected as the principal pixel 802 by the principal pixel detection module 720.

[0059] The context comparison module 730 is configured to generate a replacement label 732 for each of N original pixels, at least partially based on the corresponding transparency value 704 of the pixel. More specifically, the context comparison module 730 can compare the transparency value of the primary pixel 722 with the transparency value 704 for each of the remaining original pixels. In some embodiments, the context comparison module 730 can use the replacement label 732 to label any original pixel having a transparency value that differs from the transparency value of the primary pixel 722 by at least a threshold amount. (See, for example, reference...) Figure 8B The diagram shows that each of the black pixels P0-P3 has the same transparency value (e.g., α=255). More specifically, the first three pixels P0-P2 have the same transparency value as the major pixel P3. Conversely, each of the white pixels P4-P7 has a transparency value that differs from the major pixel P3 by up to one bit (α=254). Figure 8B In this embodiment, a 1-bit difference is a threshold used to receive the replacement label. Therefore, the context comparison module 730 can generate an "empty" label (e.g., label = 0) for each of the black pixels P0-P3 and a "replacement" label (e.g., label = 1) for each of the white pixels P4-P7.

[0060] Pixel replacement module 740 is configured to generate updated pixel data 706 by selectively changing pixel values ​​for one or more of N original pixels, at least in part based on replacement label 732. More specifically, pixel replacement module 740 can change the pixel value for any of the N pixels labeled as replacements (e.g., the labeled pixels). In some embodiments, pixel replacement module 740 can replace the color information 705 of the labeled pixels with color information for the dominant pixel 722. (See, for example, reference...) Figure 8C The color information of the primary pixel P3 is identified as the replacement pixel value 803. Since each of pixels P4-P7 has been labeled as a replacement by the context comparison module 730, the pixel replacement module 740 can replace the color information of pixels P4-P7 with the replacement pixel value 803. Figure 8C In one embodiment, the pixel replacement process causes each of pixels P4-P7 to change from a white pixel to a black pixel (e.g., the color of the primary pixel P3).

[0061] Pixel adjustments performed by the context-aware pixel adjustment circuit 700 can affect the color of interpolated pixels in a scaled image. For example, as... Figure 8AAs shown, the four original pixels to the left of the interpolated pixel position 801 are black, while the four original pixels to the right of the interpolated pixel position 801 are white. Therefore, without any pixel adjustment, based on an 8-tap interpolation of the four black pixels P0-P3 and the four white pixels P4-P7, the color of the interpolated pixel at pixel position 801 will be a different color from any of the original pixels P0-P7 (e.g., a shade of gray). This could manifest as ringing, blurring, and / or fading at the boundaries between black and white pixels in a scaled image. However, as... Figure 8C As shown, after pixel adjustment, all eight updated pixels P0-P7 are black. Consequently, the interpolated pixel at pixel position 801 will also be black. Therefore, this embodiment can produce sharper and / or more defined edges at the boundaries between black and white pixels in a scaled image.

[0062] Figure 9 Another block diagram of an image scaling circuit 900 according to some embodiments is shown. The image scaling circuit 900 may be... Figure 4 Image scaler 420 and / or Figure 6 One embodiment of the image scaling circuit 900. The image scaling circuit 900 includes an image data interface 910, a processor 920, and a memory 930.

[0063] Image data interface 910 can be used to communicate with one or more image sources and / or display devices coupled to image scaling circuitry 900. Example image sources may include, but are not limited to, image capture devices, graphics generators, image encoders, and / or other processing resources. Example display devices may include, but are not limited to, light-emitting diode (LED), organic LED (OLED), cathode ray tube (CRT), liquid crystal display (LCD), plasma and electroluminescent (EL) displays. In some embodiments, image data interface 910 may be configured to receive raw image data from one or more image sources and output a scaled version of the image data to one or more display devices.

[0064] The memory 930 may include image data storage 931, configured to store raw image data received via the image data interface 910 and / or interpolated image data output via the image data interface 910. The memory 930 may also include a non-transitory computer-readable medium (e.g., one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, hard disk drive, etc.), which may store at least the following software (SW) modules:

[0065] ●Context-aware (CA) pixel adjustment SW module 932, for updating received image data for interpolation purposes by selectively changing pixel values ​​for one or more of the original pixels, the CA pixel adjustment SW module 932 also includes:

[0066] ○ The principal pixel detection submodule 933 is used to select a principal pixel from the number of original pixels (N) based at least in part on the pixel position of the interpolation associated with N original pixels;

[0067] ○ Context comparison submodule 934, for generating a replacement label for each of the N original pixels, based at least in part on the transparency value of the major pixel and the corresponding transparency value for each of the remaining original pixels; and

[0068] ○ Pixel replacement submodule 935, configured to selectively change the color information of one or more of N original pixels, based at least in part on the color information of the primary pixel and the corresponding replacement label for each of the remaining original pixels; and

[0069] ●Pixel interpolation SW module 936, for generating interpolated image data based on updated image data, for example, by interpolating each pixel value of the interpolated image data from N pixel values ​​of the updated image data.

[0070] Each software module includes instructions that, when executed by processor 920, cause image scaling circuitry 900 to perform a corresponding function. The non-transitory computer-readable medium of memory 930 therefore includes instructions for performing the following references. Figure 10 and 11 The instructions described, including all or part of the operations.

[0071] Processor 920 may be any suitable processor or one or more processors capable of executing scripts or instructions of one or more software programs stored in image scaling circuitry 900. For example, processor 920 may execute CA pixel adjustment SW module 932 to generate updated image data for interpolation purposes by selectively changing the pixel values ​​of one or more original pixels of the received image data. Processor 920 may also execute pixel interpolation SW module 936 to generate interpolated image data based on the updated image data, for example, by interpolating each pixel value of the interpolated image data from N pixel values ​​of the updated image data.

[0072] When executing the CA pixel adjustment SW module 932, the processor 920 may also execute a principal pixel detection submodule 933, a context comparison submodule 934, and / or a pixel replacement submodule 935. For example, the processor 920 may execute the principal pixel detection submodule 933 to select a principal pixel from the number (N) of original pixels based at least in part on the pixel position of the interpolated pixel associated with the N original pixels. The processor 920 may also execute the context comparison submodule 934 to generate a replacement label for each of the N original pixels based at least in part on the transparency value of the principal pixel and the corresponding transparency value for each of the remaining original pixels. Furthermore, the processor 920 may execute the pixel replacement submodule 935 to selectively change the color information of one or more of the N original pixels based at least in part on the color information of the principal pixel and the corresponding replacement label for each of the remaining original pixels.

[0073] Figure 10 This is an illustrative flowchart depicting an example image processing operation 1000 according to some embodiments. (See, for example, references...) Figure 6 Operation 1000 can be performed by the image scaling circuit 600 to convert the received image data 602 into interpolated image data 606, wherein the interpolated image data 606 represents a scaled version of the received image data 602.

[0074] The image scaling circuit 600 can receive image data (1010) for a plurality of pixels corresponding to the first image. For example, the first image may have a native resolution different from the resolution of the display area on which the image is to be presented. Therefore, the first image may need to be scaled (e.g., resized) to fit the resolution of the display. In some respects, the first image may be upscaled to a higher resolution. In some other respects, the first image may be downscaled to a lower resolution.

[0075] Image scaling circuitry 600 can update image data (1020) by selectively altering color information for one or more of the pixels, at least partially based on pixel-based transparency values. For example, transparency values ​​can provide context for one or more objects in the received image data. (See also: Regarding...) Figure 4 and 5 As described above, the edges of an object can be identified by changes in the transparency values ​​between pixels associated with the object and pixels outside the object (e.g., exceeding a threshold amount). In some embodiments, the image scaling circuit 600 can modify the color information for one or more pixels adjacent to the edges of the object, such that color information associated only with the object in the original image can be used to interpolate the color information of the object in the scaled image.

[0076] The image scaling circuit 600 can then generate an interpolated image (1030) based on the updated image data. For example, the interpolated image may correspond to a scaled or resized version of the received image. In some embodiments, the image scaling circuit 600 can interpolate the color information for each pixel of the interpolated image from color information for the number (N) of pixels associated with the updated image data (e.g., N-tap interpolation). Suitable pixel interpolation techniques may include, but are not limited to, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc. Thus, the color of each pixel of the interpolated image may depend on a weighted average of the colors of each of the N pixels associated with the updated image data. As a result, operation 1000 can prevent or reduce artifacts along the edges of objects in the scaled image.

[0077] Figure 11 This is an illustrative flowchart depicting an example context-aware pixel adjustment operation 1100 according to some embodiments. (See, for example, references...) Figure 7 Operation 1100 can be performed by the context-aware pixel adjustment circuit 700 to selectively change the pixel values ​​of one or more pixels in the original image for the purpose of generating interpolated pixels for a scaled image. More specifically, the interpolated pixels can be derived from the number (N) of pixels in the original image.

[0078] The pixel adjustment circuit 700 can determine the interpolated pixel positions (1110) associated with N original pixels. The interpolated pixel positions can correspond to the positions of the interpolated pixels in the scaled image. In some embodiments, the pixel adjustment circuit 700 can determine the interpolated pixel positions based at least in part on scaling information. For example, the scaling information can include the scaling ratio to be performed on the original image and the initial phase (e.g., to produce a scaled image).

[0079] The pixel adjustment circuit 700 can identify the dominant pixel (1120) among N original pixels based on the interpolated pixel position. For example, the dominant pixel could be the pixel in the original image that is geographically closest to the interpolated pixel position. This can be achieved, for example, by referring to... Figure 8A The interpolated pixel position 801 is located between pixels P3 and P4. However, the interpolated pixel position 801 is closer to the position of the black pixel P3 than the white pixel P4. Therefore, the pixel adjustment circuit 700 can select the black pixel P3 as the primary pixel 802.

[0080] The pixel adjustment circuit 700 can also label each pixel (1130) at least in part based on the transparency value of the primary pixel. In some embodiments, the pixel adjustment circuit 700 can label any original pixel having a transparency value that differs from the transparency value of the primary pixel by at least a threshold amount. (See, for example, reference...) Figure 8BEach of the white pixels P4-P7 has a transparency value that differs from the transparency value of the major pixel P3 by a threshold amount (e.g., 1 bit). Therefore, the pixel adjustment circuit 700 can label each of the white pixels P4-P7 for replacement (e.g., label = 1).

[0081] The pixel adjustment circuit 700 can then modify the color information (1140) for any original pixel that has been labeled as a replacement. In some embodiments, the pixel adjustment circuit 700 can replace the color information of the labeled pixel with the color information for the primary pixel. (See, for example, reference...) Figure 8C The color information of the primary pixel P3 is identified as the replacement pixel value 803. Since each of pixels P4-P7 has been labeled for replacement, the pixel adjustment circuit 700 can replace the color information of pixels P4-P7 with the replacement pixel value 803. As a result, operation 1100 changes the color of each of pixels P4-P7 from white to black for the purpose of deriving the color of the interpolated pixels.

[0082] Those skilled in the art will appreciate that information and signals can be represented using any of a variety of different techniques and skills. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or particles, light fields or particles, or any combination thereof.

[0083] Furthermore, those skilled in the art will appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in connection with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in accordance with their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in varying ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure.

[0084] The methods, sequences, or algorithms described in connection with the aspects disclosed herein can be implemented directly in hardware, in a software module executed by a processor, or a combination of both. The software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be integrated into the processor.

[0085] In the foregoing description, embodiments have been described with reference to specific examples. However, it will be apparent that various modifications and changes can be made thereto without departing from the broader scope of this disclosure as set forth in the appended claims. Therefore, the description and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A method of image processing, comprising: receiving image data for a plurality of pixels corresponding to a first image, the image data including color information and an opacity value for each of the plurality of pixels; determining a boundary of the first image based on the opacity values, the boundary delineating a first region of the first image from a second region of the first image; updating the image data by selectively altering the color information for one or more of the pixels within the first region based at least in part on color information of one or more of the pixels within the second region; and interpolating a pixel within the second region based on the updated image data.

2. The method of claim 1, wherein the interpolated pixel represents a scaled version of the first image.

3. The method of claim 1, further comprising: determining context information about the first image based at least in part on the opacity values.

4. The method of claim 1, wherein the updating comprises: comparing the opacity value for a first pixel of the plurality of pixels to a target opacity value; and altering the color information of the first pixel when the opacity value for the first pixel differs from the target opacity value by at least a threshold amount.

5. The method of claim 4, further comprising: determining an interpolated pixel position based on a number of neighboring pixels of the plurality of pixels, the neighboring pixels including at least the first pixel; and selecting the target opacity value based at least in part on the interpolated pixel position, wherein the interpolated pixel position corresponds to a position of a pixel in the interpolated pixel.

6. The method of claim 5, wherein the selecting comprises: identifying the neighboring pixel that is closest to the interpolated pixel position; and selecting the target opacity value based on the opacity value for the identified pixel.

7. The method of claim 6, wherein the altering comprises: altering the color information of the first pixel to the color information of the identified pixel.

8. The method of claim 6, wherein the updating further comprises: comparing the opacity value for each of the neighboring pixels to the opacity value for the identified pixel; tagging any of the neighboring pixels that have an opacity value that differs from the opacity value of the identified pixel by at least the threshold amount; and altering the color information of the tagged neighboring pixels to the color information of the identified pixel.

9. The method of claim 8, wherein the interpolating comprises: interpolating the pixel of the interpolated pixel using the color information for the neighboring pixels after the updating.

10. An image scaling circuit, comprising: a processing system; and a memory storing instructions that, when executed by the processing system, cause the image scaling circuit to: receive image data for a plurality of pixels corresponding to a first image, the image data including color information and an opacity value for each of the plurality of pixels; determine a boundary of the first image based on the opacity values, the boundary delineating a first region of the first image from a second region of the first image; update the image data by selectively altering the color information for one or more of the pixels within the first region based at least in part on color information of one or more of the pixels within the second region; and interpolate a pixel within the second region based on the updated image data. determine a boundary of the first image based on the transparency values, the boundary delineating a first region of the first image from a second region of the first image; update the image data by selectively changing color information for one or more of the pixels within the first region based at least in part on color information of one or more of the pixels within the second region; and interpolate pixels within the second region based on the updated image data.

11. The image scaling circuit of claim 10, wherein execution of the instructions to update the image data causes the graphics scaling circuit to: compare the transparency value for a first pixel of the plurality of pixels to a target transparency value; and change the color information of the first pixel when the transparency value for the first pixel differs from the target transparency value by at least a threshold amount.

12. The image scaling circuit of claim 11, wherein execution of the instructions further causes the graphics scaling circuit to: determine an interpolated pixel position based on a number of neighboring pixels of the plurality of pixels, the neighboring pixels including at least the first pixel; and select the target transparency value based at least in part on the interpolated pixel position, wherein the interpolated pixel position corresponds to a position of a pixel in the interpolated pixel.

13. The image scaling circuit of claim 12, wherein execution of the instructions to select the target transparency value causes the graphics scaling circuit to: identify the neighboring pixels that are closest to the interpolated pixel position; and select the target transparency value based on the transparency value for the identified pixels.

14. The image scaling circuit of claim 13, wherein execution of the instructions to change the color information of the first pixel causes the graphics scaling circuit to: change the color information of the first pixel to the color information of the identified pixels.

15. The image scaling circuit of claim 13, wherein execution of the instructions to update the image data further causes the graphics scaling circuit to: compare the transparency value for each of the neighboring pixels to the transparency value for the identified pixels; tag any of the neighboring pixels that have a transparency value that differs from the transparency value of the identified pixels by at least the threshold amount; and change the color information of the tagged neighboring pixels to the color information of the identified pixels.

16. The image scaling circuit of claim 12, wherein execution of the instructions to interpolate pixels within the second region causes the graphics scaling circuit to: interpolate the color information for each of the neighboring pixels after the update.

17. An image processing system comprising: encoding circuitry configured to: receive first image data from a first image source; receive second image data from a second image source; and ​ generating third image data based on the first image data and the second image data, the third image data including color information and a transparency value for each of a plurality of pixels corresponding to a third image; and scaling circuitry configured to: determine a boundary of the third image based on the transparency value, wherein the boundary delineates a first region of the third image and a second region of the third image, and wherein the first region corresponds to the first image data and the second region corresponds to the second image data; update the third image data by selectively altering the color information for one or more of the pixels within the first region based at least in part on color information of one or more of the pixels within the second region; and interpolate a pixel within the second region based on the updated third image data.

18. The image processing system of claim 17, wherein a first subset of the pixels are derived from the first image data and a second subset of the pixels are derived from the second image data, and wherein the transparency value for the first subset of the pixels differs from the transparency value for the second subset of the pixels by at least a threshold amount.

19. The image processing system of claim 18, wherein the scaling circuitry is further configured to: determining an interpolated pixel position based on a number of neighboring pixels of the plurality of pixels, wherein the interpolated pixel position corresponds to a position of a pixel in the interpolated pixel; and identify the neighboring pixels that are closest to the interpolated pixel location; and select a target transparency value based on the transparency value for the identified pixel.

20. The image processing system of claim 19, wherein the scaling circuitry updates the image data by: comparing the transparency value for each of the neighboring pixels to the transparency value for the identified pixel; tagging any of the neighboring pixels that have a transparency value that differs from the transparency value of the identified pixel by at least the threshold amount; and altering the color information of the tagged neighboring pixels to the color information of the identified pixel.

Images