Image mixing mode system and method

Abstract

An apparatus (10) may include an electronic display (12) for displaying an image frame based on mixed image data (60) and image processing circuitry (28) for generating the mixed image data (60) by combining first image data (e.g., 64) and second image data (e.g., 62) via a mixing operation (52). The blending operation (52) may include receiving graphics alpha data (94) indicative of a transparency factor to be applied to the first image data (e.g., 64) to generate a first layer of the blending operation (52). The mixing operation (52) may also include superimposing the first layer onto a second layer based on the second image data (e.g., 62). Superposing the first layer onto the second layer may include adding a first pixel value of the first image data (e.g., 64) that includes a negative pixel value and is enhanced by the transparency factor to a second pixel value of the second image data (e.g., 62) to generate a mixed pixel value of the mixed image data (60).

Description

Background Art

[0001] The present disclosure generally relates to display image processing and, more particularly, to blending images from multiple sources into a single image.

[0002] This section is intended to introduce to the reader various aspects of the art that may be related to various aspects of the present disclosure, which are described and / or claimed hereinafter. This discussion is believed to be helpful to provide the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light and not as an admission of prior art.

[0003] Electronic devices typically use one or more electronic displays to present visual information, such as text, still images, and / or video, by displaying one or more images. For example, such electronic devices may include computers, mobile phones, portable media devices, tablets, televisions, virtual reality headsets, and vehicle dashboards. To display an image, an electronic display may control the light emission of its display pixels based at least in part on corresponding image data. Additionally, the image data may be processed to account for one or more physical or digital effects associated with displaying the image data. For example, the image data may be compensated for pixel aging (e.g., aging compensation), crosstalk between electrodes within the electronic device, transitions from previously displayed image data (e.g., pixel drive compensation), warping, contrast control, and / or other factors that may cause a viewer to perceive distortion or artifacts.

[0004] In some scenarios, such as in virtual reality, mixed reality, and / or augmented reality, it may be desirable to combine image data from multiple sources into an image frame. For example, generated objects may be combined into an image capture of a real-life environment (e.g., via a camera). However, combining image data from multiple sources together may sometimes result in the generated image looking unnatural or out of place. Summary of the Invention

[0005] A summary of certain embodiments disclosed herein is set forth below. It should be understood that presenting these aspects is merely to provide the reader with a concise summary of these particular embodiments and that these aspects are not intended to limit the scope of the present disclosure. Indeed, the present disclosure may cover a variety of aspects that may not be set forth below.

[0006] According to embodiments of the present disclosure, image blending (e.g., for virtual reality, mixed reality, and / or augmented reality) can be utilized to combine image data from multiple sources into a single image frame. For example, generated objects can be combined into an image capture of a real-life environment (e.g., via a camera), a portion of the captured image can be combined into a virtual environment, and / or a combination of both. As such, image data from multiple sources can be blended together to form a single output image.

[0007] In some embodiments, an image processing circuit can perform blending in a warped image space. For example, the image data of each source can be rendered and / or transformed into a common image space. The warping process can provide compensation for environmental effects, the nature of the display, the viewer's viewpoint vision, image processing warping (such as shifting and scaling), and / or other distortion factors. Additionally, different blending modes can be implemented, such as differential rendering (DR) blending that can consider shadow effects, matte blending, and / or a combined blending mode that utilizes aspects of both matte blending and DR blending.

[0008] For example, in the matte blending mode, matte data can be generated based on a set of captured image data (e.g., camera image data) to indicate which part(s) of the captured image data are to be utilized in the blending. In other words, the matte data can be used to distinguish the image data layers to be stacked or otherwise combined during the blending. For example, a camera feed of image data (e.g., captured image data corresponding to one or more color components, such as red image data, green image data, and blue image data (RGB image data)) can show a person's hand in front of them. Matte data can be generated to outline / represent the portion of the image data of the camera feed associated with the hand or other desired content. Then, the matte data can be combined with the image data of the camera feed (e.g., RGB image data) to form matte alpha data. This matte alpha data can represent the portion of the RGB image data of the image to be used in the blending with opacity information (e.g., alpha), which defines which part of the captured image data is visible after the blending. Then, the matte alpha data can be used to combine the captured image data with another set of input image data (e.g., graphic image data) (e.g., via overlay). For example, layers can be assigned to different sets of image data, and the pixel values of the captured image data can be overlaid on another set of input image data (e.g., graphic image data) based on the matte alpha data (e.g., to replace or sum with the other set of input image data).

[0009] Additionally or alternatively, the DR blend may include generating a set of graphical image data that includes negative pixel values (e.g., negative color) and graphical alpha data that (e.g., via opacity information) distinguishes portions of the graphical image data to be overlaid with other image data. For example, graphical image data may be generated to represent an object and a projected shadow of the object. The projected shadow may correspond to negative image data that is added to other image data (e.g., captured image data) based on the graphical alpha data when blended with the other image data. The graphical alpha data may indicate which portion of the graphical image data and / or how the graphical image data is added to the other image data. Additionally, in some embodiments, the DR blend and the matte blend may be performed together to generate multiple graphical layers and / or captured image data to be blended for a single image frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Aspects of the present disclosure may be better understood when the following detailed description is read in conjunction with the accompanying drawings, in which:

[0011] Figure 1 is a schematic diagram of an electronic device including an electronic display, according to an embodiment;

[0012] Figure 2 is an example of an electronic device in the form of a handheld device, according to an embodiment Figure 1 of;

[0013] Figure 3 is another example of an electronic device in the form of a tablet device, according to an embodiment Figure 1 of;

[0014] Figure 4 is another example of an electronic device in the form of a computer, according to an embodiment Figure 1 of;

[0015] Figure 5 is another example of an electronic device in the form of a watch, according to an embodiment Figure 1 of;

[0016] Figure 6 is another example of an electronic device in the form of a laptop, according to an embodiment Figure 1 of;

[0017] Figure 7 is a schematic diagram of an image processing circuit including a warping block and a blending block, according to an embodiment Figure 1 of;

[0018] Figure 8 is a schematic diagram of a portion of an image processing circuit, according to an embodiment Figure 7 of;

[0019] Figure 9 Schematic diagram of example captured image data and graphic image data mixed via matte compositing according to an embodiment;

[0020] Figure 10 According to an embodiment Figure 7 and Figure 8 Schematic diagram of a part of a mixing block according to an embodiment;

[0021] Figure 11 Schematic diagram of example captured image data and graphic image data mixed via differential rendering compositing according to an embodiment;

[0022] Figure 12 Schematic diagram of example captured image data and graphic image data mixed via combined compositing according to an embodiment;

[0023] Figure 13 Schematic diagram of example captured image data and graphic image data mixed via multi - layer combined compositing according to an embodiment; and

[0024] Figure 14 Flowchart of an example process using matte compositing, differential rendering compositing, or combined compositing according to an embodiment. DETAILED DESCRIPTION

[0025] One or more specific embodiments of the present disclosure will be described below. These described embodiments are merely examples of the currently disclosed technology. Additionally, an attempt is made to provide a brief description of these embodiments, and not all features of an actual specific implementation may be described in this specification. It should be understood that in the development of any such actual specific implementation, as in any engineering or design project, specific decisions specific to many specific implementations must be made to achieve the developer's specific goals, such as meeting system - related and business - related constraints that can vary from one specific implementation to another. Moreover, it should be appreciated that such development efforts can be complex and time - consuming, but for those of ordinary skill in the art who benefit from the present disclosure, such development efforts can still be routine tasks of design, preparation, and manufacturing.

[0026] When introducing elements of various embodiments of the present disclosure, the articles "a", "an", and "the" are intended to mean that there is one or more of the elements. The terms "comprising", "including", and "having" are intended to be inclusive and mean that there may be additional elements other than the listed elements. Additionally, it should be understood that a reference to "one embodiment" or "an embodiment" of the present disclosure is not intended to be construed as excluding the existence of additional embodiments that also incorporate the recited features. Further, the phrase A "based on" B is intended to mean that A is at least partially based on B. Moreover, the term "or" is intended to be inclusive (e.g., logical or) and not exclusive (e.g., logical xor). In other words, the phrase A "or" B is intended to mean A, B, or both A and B.

[0027] Electronic devices typically use electronic displays to present visual information. Such electronic devices may include computers, mobile phones, portable media devices, tablets, televisions, virtual reality headsets, and vehicle dashboards, among others. To display an image, an electronic display controls the brightness (and thus the color) of its display pixels based on corresponding image data received at a particular resolution. For example, an image data source may provide image data as a pixel data stream, where the data for each pixel indicates the target brightness (e.g., luminance and / or color) of one or more display pixels located at the corresponding pixel position. In some embodiments, the image data may indicate the brightness of each color component, for example, via red component image data, blue component image data, and green component image data (collectively referred to as RGB image data (e.g., RGB, sRGB)). Additionally or alternatively, the image data may be indicated by a luminance channel and one or more chrominance channels (e.g., YCbCr, YUV, etc.), grayscale (e.g., gray levels), or other color bases. It should be understood that, as disclosed herein, the luminance channel may encompass linear luminance values, non-linear luminance values, and / or gamma-corrected luminance values.

[0028] Additionally, the image data may be processed to account for one or more physical or digital effects associated with displaying the image data. For example, the image data may be compensated for pixel aging (e.g., aging compensation), crosstalk between electrodes within the electronic device, transitions from previously displayed image data (e.g., pixel drive compensation), warping, contrast control, and / or other factors that may cause a viewer to perceive distortion or artifacts.

[0029] Additionally, image blending (e.g., for virtual reality, mixed reality, and / or augmented reality) can be utilized to combine image data from multiple sources into a single image frame. For example, a generated object can be combined into an image capture of a real-life environment (e.g., via a camera), a portion of the captured image can be combined into a virtual environment, and / or a combination of both. Accordingly, image data from multiple sources can be blended together to form a single output image.

[0030] The multiple image data sources can include captured image data (e.g., via a camera), generated (e.g., via one or more processors) image data, and / or received, stored, or otherwise predefined image data. It should be understood that the sources of image data to be blended are given as non-limiting examples, and any suitable portion of the image data can be blended into a single image frame. Additionally, in some embodiments, one or more sources of image data can provide real-time image data (e.g., a camera feed, a continuously processed graphics feed, etc.), and the blending can be performed in real time such that real-world inputs (e.g., a camera feed and / or input / output devices) can affect the received image data.

[0031] In some embodiments, the image processing circuitry can perform the blending in a warped image space. For example, the image data of each source can be rendered in the warped image space and / or transformed into the warped image space. The warping process can provide compensation for environmental effects, the nature of the display, the viewer's viewpoint perspective, image processing warping (such as shifting and scaling), and / or other distortion factors. For example, the image data from a camera can be warped to compensate for lens distortion associated with the captured image and / or to compensate for any lens effects of the display panel. It should be understood that the image data from multiple sources can be blended in any suitable common image space. For example, the image data from each source can be rendered in the display image space or transformed (e.g., warped) into the display image space such that the image data is not warped after blending, or the image data can be blended in an intermediate warped space and warped into the display image space after blending.

[0032] In addition, different blending modes can be implemented, such as differential rendering (DR) blending, matte blending, and / or a combined blending mode that utilizes aspects of both matte blending and DR blending. For example, in matte blending mode, matte data can be generated based on a set of input image data (e.g., captured image data) to indicate which part(s) of the input image data are to be utilized in the blend. In other words, the matte data can be used to distinguish the image data layers that are to be "stacked" (e.g., overlaid and / or combined) during the blend. For example, a camera feed of image data (e.g., image data corresponding to one or more color components, such as red image data, green image data, and blue image data (RGB image data)) can show a person's hand in front of them. Matte data can be generated to outline / represent the part of the camera feed image data associated with the hand. Then, the matte data can be combined with the camera feed image data (e.g., RGB image data) to form matte alpha data. This matte alpha data can represent the RGB image data part of the image to be used in the blend with opacity information (e.g., alpha). Then, the matte alpha data can be used to combine the camera feed image data with another set of input image data (e.g., graphic image data) (e.g., via overlay). For example, layers can be assigned to different sets of image data, and the pixel values of the camera feed image data can be overlaid on top of another set of input image data (e.g., graphic image data) based on the matte alpha data (e.g., to replace or sum with the other set of input image data).

[0033] To produce a realistic image with features such as shadows, DR blending can include generating a set of input image data (e.g., graphic image data) that includes negative pixel values (e.g., negative color) and graphic alpha data, which (e.g., via opacity information) distinguishes the part of the input image data (e.g., graphic image data) that is to be overlaid with other image data (e.g., captured image data or other graphic image data) and the part of the input image data that can be summed with other image data. For example, graphic image data can be generated to represent an object and its projected shadow. The projected shadow can correspond to negative image data that is summed with other image data (e.g., captured image data) when blended with that other image data to produce a darkened opacity depicting the shadow. The graphic alpha data can represent which part and / or how the graphic image data is to be summed with other image data. Additionally, in some embodiments, DR blending and matte blending can be performed together to generate multi-layer graphic and / or captured image data.

[0034] In view of the above,[ Figure 1is an example electronic device 10 having an electronic display 12 with independently controllable color component illuminators (e.g., projectors, backlights, etc.). As described in more detail below, the electronic device 10 can be any suitable electronic device, such as a computer, mobile phone, portable media device, tablet, television, virtual reality headset, wearable device (e.g., watch), vehicle dashboard, etc. Thus, it should be noted that Figure 1 is only an example of a particular specific implementation and is intended to illustrate the types of components that can be present in the electronic device 10.

[0035] The electronic device 10 can include one or more electronic displays 12, input devices 14, input / output (I / O) ports 16, a processor core complex 18 having one or more processors or processor cores, local memory 20, a main memory storage device 22, a network interface 24, a power supply 26, and image processing circuitry 28. Figure 1 The various components described herein can include hardware elements (e.g., circuits), software elements (e.g., tangible non-transitory computer-readable media storing instructions), or combinations of hardware and software elements. It should be understood that the various components can be combined into fewer components or separated into additional components. For example, the local memory 20 and the main memory storage device 22 can be included in a single component. Additionally, the image processing circuitry 28 (e.g., graphics processing unit, display image processing pipeline, etc.) can be included in the processor core complex 18 or implemented separately.

[0036] The processor core complex 18 is operatively coupled to the local memory 20 and the main memory storage device 22. Thus, the processor core complex 18 can execute instructions stored in the local memory 20 or the main memory storage device 22 to perform operations such as generating and / or transmitting image data for display on the electronic display 12. Accordingly, the processor core complex 18 can include one or more general-purpose microprocessors, one or more application-specific integrated circuits (ASICs), one or more field-programmable logic arrays (FPGAs), or any combination thereof.

[0037] In addition to program instructions, the local memory 20 or the main memory storage device 22 can store data to be processed by the processor core complex 18. Thus, the local memory 20 and / or the main memory storage device 22 can include one or more tangible non-transitory computer-readable media. For example, the local memory 20 can include random access memory (RAM), and the main memory storage device 22 can include read-only memory (ROM), rewritable non-volatile memory such as flash memory, hard disk drive, optical disc, etc.

[0038] The network interface 24 can transfer data with another electronic device or network. For example, the network interface 24 (such as a radio frequency system) can enable the electronic device 10 to be communicatively coupled to a personal area network (PAN) (such as a Bluetooth network), a local area network (LAN) (such as an 802.11x Wi-Fi network), and / or a wide area network (WAN) (such as a 4G, Long Term Evolution (LTE), or 5G cellular network).

[0039] The power supply 26 can provide power to operate the processor core complex 18 and / or other components in the electronic device 10. Thus, the power source 26 can include any suitable energy source, such as a rechargeable lithium polymer (Li-poly) battery and / or an alternating current (AC) power converter.

[0040] The I / O port 16 can enable the electronic device 10 to interact with various other electronic devices. The input device 14 can enable a user to interact with the electronic device 10. For example, the input device 14 can include buttons, a keyboard, a mouse, a touchpad, etc. Additionally or alternatively, the electronic display 12 can include a touch-sensing component that enables user input to the electronic device 10 by detecting the occurrence and / or position of an object touching its screen (such as the surface of the electronic display 12).

[0041] The electronic display 12 can display a graphical user interface (GUI) (such as an operating system or a computer program), an application interface, text, a still image, and / or video content. The electronic display 12 can include a display panel having one or more display pixels to facilitate the display of an image. Additionally, each display pixel can represent a subpixel that controls the brightness of a color component (such as red, green, or blue). As used herein, a display pixel can refer to a set of subpixels (such as red, green, and blue subpixels) or can refer to a single subpixel.

[0042] As described above, the electronic display 12 can display an image by controlling the brightness output (such as light emission) of the subpixels based on corresponding image data. In some embodiments, the pixel or image data can be generated by an image source (such as the processor core complex 18, a graphics processing unit (GPU), or an image sensor (such as a camera)). Additionally, in some embodiments, image data can be received from another electronic device 10, for example, via the network interface 24 and / or the I / O port 16. Furthermore, in some embodiments, the electronic device 10 can include multiple electronic displays 12 and / or can perform image processing (such as via the image processing circuit 28) on one or more external electronic displays 12 connected, for example, via the network interface 24 and / or the I / O port 16.

[0043] The electronic device 10 can be any suitable electronic device. For ease of explanation, an example of a suitable electronic device 10, particularly a handheld device 10A, is shown in Figure 2 FIG. 1. In some embodiments, the handheld device 10A can be a portable phone, media player, personal data manager, handheld gaming platform, and the like. For illustrative purposes, the handheld device 10A can be a smart phone, for example, any model available from Apple Inc.

[0044] The handheld device 10A can include a housing 30 (e.g., a casing) to, for example, protect internal components from physical damage and / or shield the internal components from electromagnetic interference. The housing 30 can at least partially surround an electronic display 12. In the depicted embodiment, the electronic display 12 displays a graphical user interface (GUI) 32 having an array of icons 34. For example, when an icon 34 is selected via an input device 14 or a touch-sensing component of the electronic display 12, an application can be launched.

[0045] The input device 14 can be accessed through an opening in the housing 30. In addition, the input device 14 can enable a user to interact with the handheld device 10A. For example, the input device 14 can enable a user to activate or deactivate the handheld device 10A, navigate the user interface to a home screen, navigate the user interface to a user-configurable application screen, activate a voice recognition feature, provide volume control, and / or toggle between a vibrate mode and a ring mode. In addition, an I / O port 16 can also be opened through the housing 30. Additionally, the electronic device can include one or more cameras 36 to capture pictures or videos. In some embodiments, the cameras 36 can be used in conjunction with virtual reality or augmented reality visualizations on the electronic display 12.

[0046] Figure 3 Another example of a suitable electronic device 10 is shown, specifically a tablet device 10B. The tablet device 10B can be any model available from Apple Inc. Figure 4 Another example of a suitable electronic device 10 is shown, specifically a computer 10C. For illustrative purposes, the computer 10C can be any or model available from Apple Inc. Figure 5 Another example of a suitable electronic device 10 is shown, specifically a watch 10D. For illustrative purposes, the watch 10D can be any APPLE Model. As depicted, the tablet device 10B, computer 10C, and watch 10D each further include an electronic display 12, an input device 14, an I / O port 16, and a housing 30. The electronic display 12 can display a GUI 32. Here, the GUI 32 displays a visualization of a clock. When the visualization is selected via the input device 14 or the touch sensing component of the electronic display 12, an application can be launched, such as transforming the GUI 32 to present Figure 2 and Figure 3 the icons 34 discussed in

[0047] See Figure 6 , the computer 10E can represent Figure 1 another embodiment of the electronic device 10. The computer 10E can be any suitable computer, e.g., a desktop computer, a server, or a laptop, but can also be a standalone media player or a video game console. By way of example, the computer 10E can be an of Apple Inc. of Cupertino, California or other similar device. It should be noted that the computer 10E can also represent a personal computer (PC) of another manufacturer. A similar housing 30 can be provided to protect and enclose the internal components of the computer 10E, e.g., the electronic display 12. In some embodiments, the user of the computer 10E can use various peripheral input devices 14 (e.g., a keyboard 14A or a mouse 14B connectable to the computer 10E) to interact with the computer 10E.

[0048] As described above, the electronic display 12 can display an image at least partially based on image data. Before displaying a corresponding image on the electronic display 12, the image data can be processed, for example, via an image processing circuit 28. Generally speaking, the image processing circuit 28 can process the image data for display on one or more electronic displays 12. For example, the image processing circuit 28 can include a display pipeline, a memory-to-memory scaler and rotator (MSR) circuit, a warping compensation circuit, or additional hardware or software means for processing the image data. The image data can be processed by the image processing circuit 28 to reduce or eliminate image artifacts, compensate for one or more different software or hardware-related effects, and / or format the image data for display on one or more electronic displays 12. It should be understood that the present technology can be implemented in standalone circuitry, software, and / or firmware, and can be considered part of, separate from, and / or parallel to a display pipeline or an MSR circuit.

[0049] For purposes of illustration, Figure 7FIG. 0 shows a portion of an electronic device 10 that includes an image processing circuit 28. The image processing circuit 28 may be implemented in the electronic device 10, the electronic display 12, or a combination thereof. Thus, the image processing circuit 28 may be included in the processor core complex 18, a timing controller (TCON) in the electronic display 12, or any combination thereof. It should be understood that although image processing is discussed herein as being performed via a plurality of image data processing blocks, embodiments may include hardware or software components to perform the techniques discussed herein.

[0050] The electronic device 10 may also include an image data source 38, a display panel 40, and / or a controller 42 that communicate with the image processing circuit 28. In some embodiments, the display panel 40 of the electronic display 12 may be a reflective technology display, a liquid crystal display (LCD), or any other suitable type of display panel 40. In some embodiments, the controller 42 may control the operation of the image processing circuit 28, the image data source 38, and / or the display panel 40. To facilitate control of the operation, the controller 42 may include a controller processor 44 and / or a controller memory 46. In some embodiments, the controller processor 44 may be included in the processor core complex 18, the image processing circuit 28, a timing controller in the electronic display 12, a separate processing module, or any combination thereof, and execute instructions stored in the controller memory 46. Additionally, in some embodiments, the controller memory 46 may be included in the local memory 20, the main memory storage device 22, a separate tangible non-transitory computer-readable medium, or any combination thereof.

[0051] As such, the image processing circuit 28 may receive source image data 48 corresponding to a desired image to be displayed on the electronic display 12 from the image data source 38. The source image data 48 may indicate target characteristics (e.g., pixel data) corresponding to the desired image using any suitable source format (e.g., RGB format, αRGB format, YCbCr format, etc.). Additionally, the source image data may be fixed or floating point and may have any appropriate bit depth. Further, the source image data 48 may reside in a linear color space, a gamma-corrected color space, or any other suitable color space. As used herein, a pixel or pixel data may refer to a grouping of sub-pixels (e.g., individual color component pixels, e.g., red, green, and blue) or the sub-pixels themselves.

[0052] As described above, the image processing circuit 28 is operable to process source image data 48 received from an image data source 38. The image data source 38 may include captured images (e.g., from one or more cameras 36), images stored in a memory, graphics generated by the processor core complex 18, or a combination thereof. Additionally, the image processing circuit 28 may include one or more sets of image data processing blocks 50 (e.g., circuits, modules, or processing stages), such as a blending block 52 and a warping block 54. It should be understood that multiple other processing blocks 56 may also be incorporated into the image processing circuit 28 before and / or after the blending block 52 and / or the warping block 54, such as a pixel contrast control (PCC) block, a color management block, a dithering block, an aging compensation (BIC) block, a scaling / rotation block, etc. The image data processing blocks 50 may receive and process the source image data 48 and output display image data 58 in a format interpretable by the display panel 40 (e.g., digital format and / or resolution). Moreover, the functions (e.g., operations) performed by the image processing circuit 28 may be divided among the various image data processing blocks 50, and although the term "block" is used herein, there may or may not be a physical or logical separation among the image data processing blocks 50.

[0053] In some embodiments, the blending block 52 and the warping block 54 may process multiple sets of source image data 48 from one or more image data sources 38 to generate blended image data for a single image frame. For example, as Figure 8 shown in a portion of the image processing circuit, the blended image data 60 may be based on multiple sets of image data (e.g., captured image data 62 and graphics image data 64). Although discussed herein as captured image data 62 and graphics image data 64, it should be understood that any suitable image data source 38 may be utilized, and the blended image data 60 may be a combination of multiple sets of captured image data 62, multiple sets of graphics image data 64, or a combination of captured image data 62 and graphics image data 64. Additionally, the graphics image data 64 may be regarded as any stored, received, processed, or otherwise generated image data. For example, the graphics image data 64 may indicate virtual reality / augmented reality / mixed reality content. Moreover, the captured image data 62 may represent any real-world content, such as a captured image, object, or scene, and may indicate real-time image capture (e.g., a camera feed) or pre-captured content.

[0054] As discussed above, multiple sets of source image data 48 (e.g., captured image data 62 and graphical image data 64) can be from multiple image data sources 38. As such, the image spaces (e.g., perspective and / or profile) of different source image data 48 can also be different. For example, the captured image data 62 can include distortion associated with a camera lens, and the graphical image data 64 can be generated based on undistorted virtual content rendering. As such, the captured image data 62 and / or the graphical image data 64 can be warped (e.g., via the warp block 54) to a common image space such that the image content of different source image data 48 can be aligned and blended in the same image space. It should be understood that the warp block 54 can perform or provide one or more different compensations for any of a variety of effects and generate warped captured data 66 and warped graphical data 68. For example, the warp block 54 can compensate for environmental effects, the nature of the display, the viewer's viewpoint vision, image processing warps (such as shifting and scaling), and / or other distortion factors. In some embodiments, the captured image data 62 and the graphical image data 64 can be warped to the common image space of the display image data 58. For example, the warped captured data 66 and the warped graphical data 68 can be compensated for one or more lensing effects of the display panel 40, e.g., to counteract the shape and / or glass refraction effects of the display panel 40. Alternatively, it should be understood that the warped captured data 66 and the warped graphical data 68 can be warped to any common image space for blending. Additionally, although shown in Figure 8 to be blended after the warp block 54, additional warping processing can also be performed on the blended image data 60 after the blend block 52.

[0055] In some embodiments, the blend block 52 can utilize one or more blend modes, such as matte blend 70, differential rendering (DR) blend 72, and / or a combined blend 74 mode that utilizes aspects of both the matte blend 70 and the DR blend 72. For example, in the matte blend 70, matte data 80 can be generated based on the captured image data 62 to indicate which part(s) of the captured image data 62 will be visible in the blend, as Figure 9As shown. The matte blend 70 can be used to overlay (e.g., layer) a portion of the captured image data 62 onto the graphic image data 64, e.g., for augmented or mixed reality. In other words, the matte data 80 can be used to select a portion of the captured image data 62 to be stacked or otherwise combined during the blend. To help illustrate, an example captured image data 62A can include a person's hand. A corresponding example matte data 80A can be generated to outline / represent the portion of the example captured image data 62A associated with the hand and / or other desired content. Additionally, the image processing circuit 28 (e.g., the warp block 54 and the blend block 52) can warp the example graphic data 64A based on the example matte data 80A and blend the example graphic data with the example captured image data 62A to generate an example blended image data 60A. It should be understood that the example blended image data 60A appears distorted in the example of Figure 9 to compensate for the distortion associated with the display panel 40. However, as discussed herein, the blend block 52 can operate in any common image space. Additionally, although shown as being input into the blend block 52 in Figure 9 , the matte data 80 can be generated by the blend block 52 (e.g., in the common image space) or generated elsewhere (e.g., via one or more processors) and warped with the captured image data 62 and / or the graphic image data 64 in the warp block 54.

[0056] As Figure 10 shown in the schematic diagram of a portion of the blend block 52 in, the matte data 80 can also be combined with the captured image data 62 to form matte alpha data 82. The matte alpha data 82 can represent the RGB image data of the warped captured data 66 to be used in a blend with opacity information (e.g., alpha), which defines which portion of the warped captured data 66 is visible after the blend. The matte alpha data 82 can then be used to combine (e.g., via overlay or pixel value addition) the warped captured data 66 with another set of input image data (e.g., the warped graphic data 68). In some embodiments, the matte data 80 can include separate alpha components 86 (e.g., a red alpha component 86A, a green alpha component 86B, and a blue alpha component 86C, collectively 86) for indicating the transparency (e.g., opacity) of each component and / or a constant transparency 88. The separate alpha components 86 can be multiplied (e.g., via a multiplier 90) with their corresponding component channels in the warped captured data 66 to generate the matte alpha data 82. Additionally, in some embodiments, the constant transparency 88 can be added (e.g., via an adder 92) to each component channel, e.g., to adjust the overall transparency of the matte alpha data 82. For example, Figure 9An example matte data 80A can be combined with the captured image data 62 (e.g., before or after warping processing) to assign opacity to the hand and general transparency to the background of the captured image data 62 such that when blended, the hand can be visible over the example graphic image data 64 (e.g., as in the example blended image data 60). Additionally, layers can be assigned to different groups of image data, and the pixel values of the warped captured data 66 can be overlaid on the warped graphic data 68 based on the matte alpha data 82 (e.g., to replace or sum with the warped graphic data).

[0057] To achieve realistic shadows, the graphic image data 64 can be layered and overlaid on the captured image data in augmented or mixed reality using DR blending 72. In some embodiments, the graphic image data 64 can be generated to include negative pixel values (e.g., negative film colors) intended to be overlaid with other image data (e.g., the captured image data 62), such as to depict a shadow effect for the content to be overlaid, as shown in the example of Figure 11 as shown. Additionally, graphic alpha data 94 can be generated that differentiates how different portions of the graphic image data 64 will be overlaid or combined (e.g., summed) with other image data (e.g., the captured image data 62), such as via transparency (e.g., alpha) information. As used herein, the transparency information of the graphic alpha data 94 and / or the matte data 80 / matte alpha data 82 can be a per-pixel transparency factor that describes how the associated image data is transparent or opaque at a pixel location. Additionally, such transparency can be used to determine how the pixel values of multiple groups of image data are added during blending.

[0058] For example, graphic image data 64B can be generated to represent an object and its projected shadow. The projected shadow can correspond to negative pixel values that are summed (e.g., weighted sum) with the other image data (e.g., the captured image data 62B) based on the graphic alpha data 94B when blended with the other image data. By adding the negative pixel values of the graphic image data 64B to the other image data (e.g., the captured image data 62B) using a weighted sum (e.g., based on the transparency factor of the graphic alpha data 94B), a more realistic shadow effect can be depicted. Additionally, if the weighted sum results in a negative pixel value for the blended image data 60, in some embodiments, a lower limit can be set such that the negative values of the blended image data 60 are replaced with zero gray values.

[0059] In Figure 11In the example of, the example graphic alpha data 94B can specify that the object will be layered opaquely (e.g., superimposed on top of the captured image data 62B), and the remaining portion of the graphic image data 64 can be additively blended with the example captured image data 62B (e.g., based on the transparency information of the graphic alpha data 94B) to include a partially transparent layer on the captured image data 62B. Additionally, it should be understood that in some embodiments, pixel data representing the background (e.g., different from the object and / or projected shadow having negative pixel values) can have a net 0 effect during blending (e.g., a pixel value of 0 and / or an alpha value corresponding to full transparency). Additionally, as described above, the graphic image data 64 can be generated by the electronic device 10 (e.g., via one or more processors and / or a graphics processor). Accordingly, in some embodiments, the graphic image data 64 can be generated as pre-combined graphic alpha image data (not shown). For example, the pre-combined graphic alpha image data can include the color component pixel values (e.g., RGB pixel values) of the warped graphic data 68 combined with the graphic alpha data 94 in a manner similar to the matte alpha data 82. Accordingly, the pre-combined graphic alpha image data can be used in the DR blending 72 instead of using the warped graphic data 68 and the graphic alpha data 94 separately. It should be understood that the example blended image data 60B looks distorted in the Figure 11 example of to compensate for the distortion associated with the display panel 40. However, as discussed herein, the blending block 52 can operate in any common image space. Additionally, although in Figure 11 is shown as being input into the blending block 52, the graphic alpha data 94 can be generated by the blending block 52 (e.g., in the common image space) or generated elsewhere (e.g., via one or more processors) and warped with the captured image data 62 and / or the graphic image data 64 in the warping block 54.

[0060] Additionally, in some embodiments, the matte blending 70 and the DR blending 72 can be used together in a combined blending 74, as Figure 12is illustrated. The composite blend 74 may be performed to generate multi-layer graphics and / or captured image data to be blended for a single image frame. For example, the graphic alpha data 94C may be utilized to indicate the portion (e.g., opaque) of the graphic image data 64C to be layered on top of the captured image data 62C, and that portion is transparent (or partially transparent in the case of using negative pixel values), such that the captured image data 62C is visible. Additionally, an example matte 80C may be used to isolate a portion of the captured image data 62C that may be layered separately from the remainder of the captured image data 62C. As such, in some embodiments, the composite blend 74 may stack a portion of the warped graphic data 68 on the warped captured data 66 and stack a portion of the warped captured data 66 on the warped graphic data 68, based on the opacity information of the graphic alpha data 94 and the matte data 80 (and / or matte alpha data 82). As such, multiple content layers may be blended to provide a more realistic aggregation of the captured image data 62 and the graphic image data 64.

[0061] Additional alpha data 95 may be used to generate additional content layers, as Figure 13 in the example of. The additional alpha data 95 may be utilized in combination with the graphic alpha data 94D and / or the matte data 80D to generate an additional layer (e.g., a top layer) above the captured image data 62D and the graphic image data 64D to incorporate additional content into the blended image data 60D. Similar to the graphic alpha data 94, the additional alpha data 95 may include transparency (e.g., alpha) information such that some or all of the underlying content is visible through it. Such additional layers may be used to display text information to the user without obstructing the visual perception of the underlying content, or to highlight certain information / content by making it opaque (e.g., making the underlying content invisible).

[0062] Return Figure 10, in some scenarios, such as when using combinational blending 74, the graphic alpha data 94 and the matte alpha data 82 (or its inverse) can be combined (e.g., added or multiplied) to generate a combined alpha 96. The combined alpha 96 can represent the portions viewable in different layers of the warped captured data 66 and the warped graphic data 68, such that when applying combinational blending, the desired pixel data (e.g., RGB pixel data) is used from each set of image data, taking into account transparency (e.g., additive pixel values). Additionally, it should be understood that although specific sets of image data are discussed herein (e.g., warped captured data 66, warped graphic data 68, matte data 80, matte alpha data 82, graphic alpha data 94, etc.), the circuitry of the blending block 52 (e.g., portion 84) can operate on any suitable variants of the image data discussed, such as inverses (e.g., 1 - X), negatives (e.g., -1X), etc. Further, in some embodiments, the blending block 52 can include control logic 98 that operates and / or includes routing logic 100, such as gates, registers, and / or multiplexers.

[0063] In some embodiments, control logic 98 and / or routing logic 100 may receive control signals 102 (e.g., matte signal 102A, DR signal 102B, and / or virtual signal 102C, collectively labeled 102) to determine what type of blending to perform (e.g., matte blend 70, DR blend 72, or combined blend 74) and / or alter aspects of the blending. For example, matte signal 102A may activate and / or provide the matte blend 70 to be performed, and DR signal 102B may activate and / or provide the DR blend 72 to be performed. Matte signal 102A and DR signal 102B may be used together to activate and / or provide the combined blend 74. Additionally, in some embodiments, virtual signal 102C may be used to activate and / or implement blend overlay 104. Blend overlay 104 may indicate an overlay message (e.g., a warning or other function) that is intended to replace other captured image data 62 and / or graphical image data 64 or otherwise layer over that other captured image data and / or graphical image data. Additionally, blend block 52 may include layering and blending computation circuitry 106 to compile layers of image data (e.g., warped captured data 66, warped graphical data 68, etc.) based on graphical alpha data 94, combined alpha data 96, and / or matte alpha data 82. For example, layering and blending computation circuitry 106 may sum warped captured data 66 and warped graphical data 68 based on graphical alpha data 94, combined alpha data 96, and / or matte alpha data 82 (e.g., a weighted sum and / or weighted average based on corresponding alpha values). Further, in some embodiments, layering and blending computation circuitry 106 may receive predefined parameters 108 to be used when performing the blending. For example, such predefined parameters 108 may include, but are not limited to, background color (e.g., layer 0 color), saturation parameter, etc.

[0064] Figure 14FIG. 110 is a flow chart of an example process using matte blend 70, DR blend 72, or combined blend 74. The image processing circuit 28 may receive (e.g., from one or more image data sources 38) captured image data 62 and graphic image data 64 (process block 112). If matte blend 70 or combined blend 74 is utilized, matte data 80 may be generated based on the captured image data 62 (process block 114). Additionally, if DR blend 72 or combined blend 74 is utilized, graphic alpha data 94 may be generated based on the graphic image data 64 (process block 116). It should be understood that in some embodiments, the graphic image data 64 may be generated as pre-combined graphic alpha image data such that the information of the graphic alpha data 94 is known. Additionally, the captured image data 62, graphic image data 64, matte data 80, and / or graphic alpha data 94 may be warped to a common image space (process block 118). It should be understood that the common image space may be any suitable image space, and certain image data may be generated, captured, or output in the common image space. For example, matte data and / or graphic alpha data may be generated in the common image space such that they are not warped. Further, if matte blend 70 or combined blend 74 is utilized, the matte data 80 may be combined with the warped captured data 66 to generate matte alpha data (process block 120). Additionally, if combined blend 74 is utilized, the graphic alpha data 94 may be combined with the matte alpha data 82 to generate combined alpha data 96 (process block 122). The layers of the warped graphic data 68 and warped captured data 66 may then be combined based on the combined alpha data 96, matte alpha data 82, and / or graphic alpha data (process block 124). Thus, image data from multiple sources (e.g., captured and digitally rendered content) may be combined to provide a more interactive, fluid, realistic, and / or immersive user experience in augmented, virtual, or mixed reality.

[0065] Although the flow chart 110 is shown in a given order, in certain embodiments, the process / decision blocks may be reordered, changed, deleted, and / or occur simultaneously. Additionally, the flow chart 110 is given as an illustrative tool and other decision blocks and process blocks may also be added according to the specific implementation.

[0066] The specific embodiments described above have been shown by way of example, and it should be understood that these embodiments may admit of various modifications and alternative forms. It should also be understood that the claims are not intended to be limited to the particular forms disclosed, but are intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure.

[0067] It is well known that the use of personally identifiable information should follow privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of inadvertent or unauthorized access or use, and the nature of the authorized use should be clearly explained to the user.

[0068] The technologies presented and claimed herein are recited and applied to specific examples of a physical and tangible nature that demonstrably improve the relevant art and are thus not abstract, intangible, or purely theoretical. Additionally, if any claim appended to the end of this specification contains one or more elements designated as "means for [performing][function]..." or "step for [performing][function]...", such elements will be construed in accordance with 35 U.S.C. 112(f). However, for any claim that contains elements designated in any other way, it is intended that such elements shall not be construed in accordance with 35 U.S.C. 112(f).

Claims

1. An apparatus, the apparatus comprising: An electronic display configured to display an image frame based on blended image data; and An image processing circuit configured to generate the blended image data by combining first image data and second image data via a blending operation, wherein the blending operation includes: Receiving graphics α data indicating a transparency factor to be applied to the first image data to generate a first layer of the blending operation; and Overlaying the first layer onto a second layer, wherein the second layer is based on the second image data, and wherein overlaying the first layer onto the second layer includes adding a first pixel value of the first image data and a second pixel value of the second image data to generate a blended pixel value of the blended image data, wherein the first pixel value of the first image data is enhanced by the transparency factor, and wherein the first pixel value includes a negative pixel value.

2. The apparatus according to claim 1, wherein the transparency factor includes per-pixel transparency weighting associated with the first image data, and wherein the blending operation includes differential rendering blending.

3. The apparatus according to claim 1, wherein the blending operation includes generating matte data based on the second image data, wherein the matte data indicates a first portion of the second image data to be included in a second layer of the blending operation and a second portion of the second image data to be included in a third layer of the blending operation.

4. The apparatus according to claim 3, wherein the blending operation includes superimposing the third layer onto the first and second layers, and wherein superimposing the third layer onto the first and second layers includes using a third pixel value of the second portion of the second image data as the blended pixel value of the blended image data at a pixel position associated with the third layer.

5. The apparatus according to claim 1, wherein the first image data includes digitally rendered content.

6. The apparatus according to claim 5, wherein the second image data includes camera-captured content.

7. The apparatus according to claim 1, wherein the image processing circuit is configured to generate the graphical α data based on the first image data.

8. The apparatus according to claim 1, wherein the image processing circuit includes a hardware pipeline having a dedicated blending circuit configured to perform the blending operation.

9. The apparatus according to claim 1, wherein the image processing circuit is configured to warp the first image data, the second image data, and the graphical α data into a common image space before superimposing the first layer onto the second layer.

10. The apparatus according to claim 9, wherein the common image space includes a display image space, and wherein the display image space is compensated for a lens effect associated with the electronic display.

11. A method, the method comprising: Receiving first image data including positive and negative pixel values; Generating graphics α data indicating a transparency factor to be applied to the first image data to generate a first layer of the blending operation, wherein the transparency factor is different for the positive pixel values and the negative pixel values; Receiving second image data corresponding to a second layer of the blending operation; and Performing the blending operation by combining the first layer and the second layer, wherein combining the first layer and the second layer includes adding the positive and negative pixel values and corresponding pixel values of the second image data based on the graphics α data to generate the blended image data.

12. The method according to claim 11, the method comprising warping the first image data and the second image data into a common image space.

13. The method according to claim 11, wherein the second image data includes captured image data.

14. The method according to claim 13, the method comprising generating matte data based on the captured image data, wherein the matte data indicates a second transparency factor to be applied to the second image data to distinguish a first portion of the second image data to be included in the second layer and a second portion of the second image data to be included in the third layer of the blending operation.

15. The method according to claim 14, the method comprising combining the first layer, the second layer, and the third layer by adding the positive pixel values and the negative pixel values to the corresponding pixel values of the second image data based on the graphic alpha data and the matte data to generate the blended image data, wherein the corresponding pixel values of the second image data include the first portion of the second image data included in the second layer and the second portion of the second image data included in the third layer.

16. The method according to claim 11, wherein the first image data includes digitally rendered content, wherein the positive pixel values correspond to objects of the digitally rendered content, and wherein the negative pixel values correspond to shadow effects of the objects.

17. A method, the method comprising: Receiving first image data including captured image content; Generating matte data indicating a portion of the first image data to be included in a first layer of the blending operation; Combining the matte data and the first image data to generate matte α data including transparency information, wherein the transparency information designates the portion of the first image data as opaque; Receiving second image data including digitally rendered image content to be included in a second layer of the blending operation; and Performing the blending operation by combining the first layer and the second layer, wherein combining the first layer and the second layer includes adding a first pixel value of the first image data and a second pixel value of the second image data based on the matte α data.

18. The method according to claim 17, the method comprising generating graphic alpha data indicating a transparency factor corresponding to a portion of the second image data, wherein the blending operation includes a combined matte and differential rendering blending operation.

19. The method according to claim 18, the method comprising combining the graphic alpha data and the matte alpha data to generate combined alpha data, wherein the combined matte and differential rendering blending operation includes combining the first layer and the second layer based on the combined alpha data.

20. The method according to claim 17, wherein the captured image content includes a real-time camera feed.

21. A system, the system comprising: Apparatus for receiving first image data including captured image content; Apparatus for generating matte data indicating a portion of the first image data to be included in a first layer of the blending operation; Apparatus for combining the matte data and the first image data to generate matte α data including transparency information, wherein the transparency information designates the portion of the first image data as opaque; Apparatus for receiving second image data including digitally rendered image content to be included in a second layer of the blending operation; and Apparatus for performing the mixing operation by combining the first layer and the second layer, wherein combining the first layer and the second layer includes adding a first pixel value of the first image data and a second pixel value of the second image data based on the matte α data.

22. The system according to claim 21, the system comprising means for generating graphic alpha data indicative of a transparency factor corresponding to a portion of the second image data, wherein the blending operation comprises a combined matte and differential rendering blending operation.

23. The system according to claim 22, the system comprising means for combining the graphic alpha data and the matte alpha data to generate combined alpha data, wherein the combined matte and differential rendering blending operation comprises combining the first layer and the second layer based on the combined alpha data.

24. One or more tangible non-transitory machine-readable media, the one or more tangible non-transitory machine-readable media comprising instructions that, when executed by a processor, cause image processing to perform operations including: Receiving first image data including positive pixel values and negative pixel values; Generating graphic alpha data indicative of a transparency factor to be applied to the first image data to generate a first layer of a blending operation, wherein the transparency factor is different for the positive pixel values and the negative pixel values; Receiving second image data corresponding to a second layer of the blending operation; and Performing the blending operation by combining the first layer and the second layer, wherein combining the first layer and the second layer comprises adding the positive pixel values and the negative pixel values to corresponding pixel values of the second image data based on the graphic alpha data to generate blended image data.

25. The tangible non-transitory machine-readable media according to claim 24, wherein the operations include warping the first image data and the second image data to a common image space.

Images