Method and system for alternating mask-based and frame-based data compression

Abstract

A method displaying a reconstructed image includes receiving an image, determining an eye gaze location of a user, compressing a primary quality region of the image using a compression ratio of X → Y bits per pixel (bpp) to form a primary quality image, and compressing a secondary quality region of the image using a compression ratio of X → Z bpp, wherein Z < Y, to form a secondary quality image. The method also includes overlaying the primary quality image on the secondary quality image to form the reconstructed image and displaying the reconstructed image.

Description

METHOD AND SYSTEM FOR ALTERNATING MASK-BASED AND FRAME-BASED DATA COMPRESSIONCROSS-REFERENCES TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 567,019, filed March 19, 2024, entitled "METHOD AND SYSTEM FOR ALTERNATING MASK-BASED AND FRAME-BASED DATA COMPRESSION," the entire contents of which is hereby incorporated by reference for all purposes.BACKGROUND OF THE INVENTION

[0002] Modem computing and display technologies have facilitated the development of systems for so-called virtual reality or augmented reality experiences, wherein digitally reproduced images or portions thereof are presented to a viewer in a manner wherein they seem to be, or may be perceived as, real. A virtual reality, or VR, scenario typically involves presentation of digital or virtual image information without transparency to other actual real- world visual input; an augmented reality, or AR, scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world around the viewer.

[0003] FIG. 1 illustrates a user's view of augmented reality (AR) through an AR device. Embodiments of the present invention are applicable to virtual content produced for display in such an AR device. Referring to FIG. 1, an augmented reality scene 100 is depicted wherein a user of an AR technology sees a real -world park-like setting 106 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the AR technology also perceives that he "sees" a robot statue 110 standing upon the real -world concrete platform 120, and a cartoon-like avatar character 102 flying by, which seems to be a personification of a bumble bee, even though these elements (i.e., cartoon-like avatar character 102 and robot statue 110 do not exist in the real world). Due to the extreme complexity of the human visual perception and nervous system, it is challengingto produce a VR or AR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements amongst other virtual or real-world imagery elements.

[0004] Despite the progress made in these display technologies, there is a need in the art for improved methods and systems related to augmented reality systems, particularly, display systems.SUMMARY OF THE INVENTION

[0005] The present invention relates generally to methods and systems related to projection display systems including wearable displays. More particularly, embodiments of the present invention provide methods and systems that provide dynamic control of data compression processes. The invention is applicable to a variety of applications in computer vision and image display systems.

[0006] In a display subsystem, there exists a stream-based mobile industry processor interface (MIPI) interface that transports the data to the endpoint display. This interface usually employs a stream-based compression algorithm that is low latency and predictive based on neighbor pixels, like the Display Stream Compression (DSC) compression algorithm. These compression algorithms try to maintain a fixed frame size with fixed compression.

[0007] Some embodiments of the present invention utilize an alternating algorithm that implements a complete frame fixed compression, for example, DSC, in conjunction with a simpler, pixel run length encoding algorithm. Therefore, the simpler run length encoding only for "black" executes (e.g., only executes, when the amount of "black" pixel groups or "black" pixels exceeds a threshold amount), otherwise the system throttles back to the full frame mechanism. As a result, embodiments of the present invention throttle the compression algorithms based on the amount of "black" pixels on the screen.

[0008] Thus, embodiments of the present invention reduce the amount of data that is transmitted over a display interface (e.g., MIPI) and can also reduce the power consumption. Moreover, the embodiments described herein keep the register transfer level (RTL) design implementation small and low latency.

[0009] According to an embodiment of the present invention, a method of displaying a reconstructed image is provided. The method includes receiving an image having a pluralityof lines of pixel data, wherein each line of the plurality of lines is defined by a plurality of pixel groups and determining a number of pixels in the image that are characterized by a pixel brightness less than a predetermined brightness level. If the number of pixels is less than a threshold, the method includes forming a first compressed image by compressing the image using a first data compression algorithm and storing the first compressed image in a memory. If the number of pixels is greater than or equal to a threshold, the method includes forming a second compressed image by: forming a mask for each line of the plurality of lines: for each pixel group of the plurality of pixel groups: defining a first bit if pixels in the pixel group are characterized by pixel values less than a brightness threshold, and defining a second bit if pixels in the pixel group are characterized by pixel values greater than or equal to the brightness threshold. The method also includes providing pixel values for pixels in pixel groups having the second bit, storing the mask and the provided pixel values for each line in a memory, forming a reconstructed image by: decompressing the first compressed image or extracting the mask and the provided pixel values for each line from the memory, forming the reconstructed image using the mask and the provided pixel values for each line, and transmitting the reconstructed image to a display.

[0010] The first data compression algorithm can include DSC or VDC-X. The first bit can be "1" and the second bit can be "0". The mask can include N bits, each line can include M pixel groups, and each line can include N x M pixels. The provided pixels do not need to include pixel values for pixels in pixel groups having the first bit. Forming the reconstructed image can include, for each line: for each pixel group: defining pixel values as black pixels if the first bit is defined for the pixel group; and defining pixel values as the provided pixel values for pixels in the pixel group if the second bit is defined for the pixel group.

[0011] According to another embodiment of the present invention, a method of compressing an image is provided. The method includes determining an eye gaze location of a user, generating a foveation map based on the eye gaze location, wherein the foveation map includes a first region of the image and a second region of the image, and compressing the first region of the image using a first compression ratio and the second region of the image using a second compression ratio higher than the first compression ratio.

[0012] Determining the eye gaze location can include use of an eye tracking camera of an augmented reality device. The foveation map can include a central region and a peripheral region. The image can include virtual content generated by an augmented reality device.Compressing the first region of the image using the first compression ratio can include compressing all blocks in the first region using the first compression ratio. The first compression ratio can be (24 — 15 bits per pixel (bpp)) and the second compression ratio can be (24 —> 8 bpp). Compressing the first region of the image using the first compression ratio can include compressing all blocks in the first region using a higher quality setting than that used to compress the second region. The method can also include post-processing image content in at least one of the first region or the second region. The method can further include decoding the compressed image using the foveation map. The second region of the image can include the first region of the image. Additionally, the method can include decoding the compressed image using the foveation map to produce a decoded first region and a decoded second region and reconstructing the image by overlaying the decoded first region over the decoded second region.

[0013] According to an embodiment of the present invention, a method of displaying a reconstructed image is provided. The method includes receiving an image having a plurality of lines of pixel data, wherein each line of the plurality of lines is defined by a plurality of pixel groups and determining a number of pixels in the image that are characterized by a pixel brightness less than a predetermined brightness level. If the number of pixels is less than a threshold, the method includes forming a first compressed image by compressing the image using a first data compression algorithm; and storing the first compressed image in a memory. If the number of pixels is greater than or equal to a threshold, the method includes forming a second compressed image by: forming a mask for each line of the plurality of lines: for each pixel group of the plurality of pixel groups: defining a first bit if pixels in the pixel group are characterized by pixel values less than a brightness threshold; and defining a second bit if pixels in the pixel group are characterized by pixel values greater than or equal to the brightness threshold; providing pixel values for pixels in pixel groups having the second bit; and storing the mask and the provided pixel values for each line in a memory. The method also includes forming a reconstructed image by: decompressing the first compressed image; or extracting the mask and the provided pixel values for each line from the memory, forming the reconstructed image using the mask and the provided pixel values for each line, and transmitting the reconstructed image to a display.

[0014] The first data compression algorithm can include DSC or Video Electronics Standards Association (VESA) Display Compression (VDC-X). The first bit can be " 1 " and the second bit can be "0". The mask can include N bits, each line can include M pixelgroups, and each line can include N x M pixels. The provided pixels do not have to include pixel values for pixels in pixel groups having the first bit. Forming the reconstructed image can include, for each line: for each pixel group: defining pixel values as black pixels if the first bit is defined for the pixel group; and defining pixel values as the provided pixel values for pixels in the pixel group if the second bit is defined for the pixel group.

[0015] According to another embodiment of the present invention, a method of compressing an image is provided. The method includes determining an eye gaze location of a user and generating a foveation map based on the eye gaze location, wherein the foveation map includes a first region of the image and a second region of the image. The method also includes compressing the first region of the image using a first compression ratio and the second region of the image using a second compression ratio higher than the first compression ratio. Determining the eye gaze location can include use of an eye tracking camera of an augmented reality device. The foveation map can include a central region and a peripheral region. The image can include virtual content generated by an augmented reality device. Compressing the first region of the image using the first compression ratio can include compressing all blocks in the first region using the first compression ratio. The first compression ratio can be (24 — 15 bits per pixel (bpp)) and the second compression ratio can be (24 —> 8 bpp). Compressing the first region of the image using the first compression ratio can include compressing all blocks in the first region using a higher quality setting than that used to compress the second region. The method can also includes post-processing image content in at least one of the first region or the second region. The compression can produce a compressed image and the method can further include decoding the compressed image using the foveation map. The second region of the image can include the first region of the image. The compression can produce a compressed image and the method can further include decoding the compressed image using the foveation map to produce a decoded first region and a decoded second region and reconstructing the image by overlaying the decoded first region over the decoded second region.

[0016] Numerous benefits are achieved by way of the present invention over conventional techniques. For example, embodiments of the present invention provide an alternating data compression algorithm that implements a complete frame compression technique, for example, DSC, with a run length encoding algorithm to provide a floor for the amount of data compression, thereby improving system performance. Moreover, some embodiments of the present invention provide methods and systems that alter the compression quality based oneye tracking, thus giving the foveated regions a higher compression ratio at a loss of quality. Embodiments of the present invention are suitable for use with a MIPI interface, thereby decreasing the amount of data that is sent over MIPI to a liquid crystal on silicon (LCOS) display or a micro-light emitting diode (pLED) display. Thereby, embodiments also produce a saving in power consumption. These and other embodiments of the invention along with many of its advantages and features are described in more detail in conjunction with the text below and attached figures.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG. 1 illustrates a user's view of augmented reality (AR) through an AR device.

[0018] FIG. 2 illustrates an example of a wearable AR display system according to an embodiment of the present invention.

[0019] FIG. 3 A illustrates a compression-level obtained as a function of time, represented by successive frames, where 100% compression denotes an all black image according to an embodiment of the present invention.

[0020] FIG. 3B illustrates a histogram of frame count versus compression for a sparsity compression system implementation according to an embodiment of the present invention.

[0021] FIG. 4A illustrates a compression-level obtained as a function of time, represented by successive frames versus frequency, for both a sparsity compression system implementation and a DSC-SPARSE system implementation, according to an embodiment of the present invention.

[0022] FIG. 4B illustrates a histogram of frame count versus compression for a sparsity compression system implementation and a DSC-SPARSE system implementation according to an embodiment of the present invention.

[0023] FIG. 5 A is a simplified flowchart illustrating a method of displaying a reconstructed image using an alternating compression algorithm according to an embodiment of the present invention.

[0024] FIG. 5B is a simplified schematic diagram illustrating operation of a mask-based compression system according to an embodiment of the present invention.

[0025] FIG. 5C is a simplified schematic diagram illustrating operation of a mask-based compression system with display decompression according to an embodiment of the present invention.

[0026] FIG. 6 is a plot of PSNR (dB) as a function of compression level for several use cases according to an embodiment of the present invention.

[0027] FIG. 7 is a simplified image illustrating an image frame divided into a primary quality region and a secondary quality region according to an embodiment of the present invention.

[0028] FIG. 8 is a simplified flowchart illustrating a method of compressing an image using different compression ratios for a primary quality region and a secondary quality region, according to an embodiment of the present invention.

[0029] FIG. 9 is a simplified image illustrating an image frame divided into primary quality tiles and secondary quality tiles according to an embodiment of the present invention.

[0030] FIG. 10 is a simplified flowchart illustrating a method of compressing an image using different compression ratios for primary quality tiles and secondary quality tiles, according to an embodiment of the present invention.

[0031] FIG. 11 is a line drawing illustrating an image compressed using a single quality setting.

[0032] FIG. 12 is a line drawing illustrating a foveated image with three foveated regions according to an embodiment of the present invention.

[0033] FIG. 13 is a line drawing illustrating a foveated image with post-processing in the foveated regions according to another embodiment of the present invention.

[0034] FIG. 14 is a foveated 3D generated image with three foveated regions according to yet another embodiment of the present invention.

[0035] FIG. 15 is a line drawing illustrating an image that can be utilized in conjunction with multiple foveation maps according to an embodiment of the present invention.

[0036] FIG. 16 is a simplified flowchart illustrating a method of compressing an image according to an embodiment of the present invention.

[0037] FIG. 17 is a simplified schematic diagram illustrating a gaze-based image foveation system according to an embodiment of the present invention.

[0038] FIG. 18 is a simplified block diagram illustrating components of an AR system according to an embodiment of the present invention.DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

[0039] The present invention relates generally to methods and systems related to projection display systems including wearable displays. More particularly, embodiments of the present invention provide methods and systems that provide dynamic control of data compression processes. The invention is applicable to a variety of applications in computer vision and image display systems.

[0040] FIG. 2 illustrates an example of wearable AR display system 200 according to an embodiment of the present invention. As shown in FIG. 2, display system 200 includes a display 202 and various mechanical and electronic modules and systems to support the functioning of display 202. Display 202 may be coupled to a frame 280, which is wearable by a display system user 290 (also referred to as a viewer) and which is configured to position the display 202 in front of the eyes of the user 290. Display 202 may be considered eyewear in some embodiments. In some embodiments, a speaker 205 is coupled to frame 280 and configured to be positioned adjacent the ear canal of the user 290 (in some embodiments, another speaker, not shown, may optionally be positioned adjacent the other ear canal of the user to provide stereo / shapeable sound control). The display system 200 may also include one or more microphones 210 or other devices to detect sound. In some embodiments, the one or more microphones 210 are configured to allow the user to provide inputs or commands to display system 200 (e.g., the selection of voice menu commands, natural language questions, etc.), and / or may allow audio communication with other persons (e.g., with other users of similar display systems). One of the one or more microphones may further be configured as a peripheral sensor to collect audio data (e.g., sounds from the user and / or environment). In some embodiments, display system 200 may further include one or more outwardly directed environmental sensors configured to detect objects, stimuli, people, animals, locations, or other aspects of the world around the user. For example, environmental sensors may include one or more cameras, which may be located, for example, facing outward so as to capture images similar to at least a portion of an ordinary field of view ofuser 290. In some embodiments, display system 200 may also include a peripheral sensor 220a, which may be separate from frame 280 and attached to the body of user 290 (e.g., on the head, torso, an extremity, etc. of user 290). Peripheral sensor 220a may be configured to acquire data characterizing a physiological state of user 290 in some embodiments. For example, peripheral sensor 220a may be an electrode.

[0041] With continued reference to FIG. 2, display 202 is operatively coupled by communications link 230, such as by a wired lead or wireless connectivity, to a local processing and data module 240 which may be mounted in a variety of configurations, such as fixedly attached to frame 280, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to user 290 (e.g., in a backpackstyle configuration, in a belt-coupling style configuration). Similarly, peripheral sensor 220a may be operatively coupled by communications link 220b, e.g., a wired lead or wireless connectivity, to local processing and data module 240. Local processing and data module 240 may comprise a hardware processor, as well as digital memory, such as non-volatile memory (e.g., flash memory or hard disk drives), both of which may be utilized to assist in the processing, caching, and storage of data. Optionally, local processing and data module 240 may include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, and so on. The data may include data a) captured from sensors (which may be, e.g., operatively coupled to frame 280 or otherwise attached to user 290), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, gyros, and / or other sensors disclosed herein; and / or b) acquired and / or processed using remote processing module 250 and / or remote data repository 260 (including data relating to virtual content), possibly for passage to display 202 after such processing or retrieval. Local processing and data module 240 may be operatively coupled by communication links 270, 275, such as via a wired or wireless communication links, to remote processing module 250 and remote data repository 260 such that these remote modules are operatively coupled to each other and available as resources to local processing and data module 240. In some embodiments, local processing and data module 240 may include one or more of the image capture devices, microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and / or gyroscopes. In some other embodiments, one or more of these sensors may be attached to frame 280, or may be standalone structures that communicate with local processing and data module 240 by wired or wireless communication pathways.

[0042] With continued reference to FIG. 2, in some embodiments, remote processing module 250 may comprise one or more processors configured to analyze and process data and / or image information, for instance including one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, and so on. In some embodiments, remote data repository 260 may comprise a digital data storage facility, which may be available through the internet or other networking configuration in a "cloud" resource configuration. In some embodiments, remote data repository 260 may include one or more remote servers, which provide information, e.g., information for generating augmented reality content, to local processing and data module 240 and / or remote processing module 250. In some embodiments, all data is stored and all computations are performed in the local processing and data module, allowing fully autonomous use from a remote module. Optionally, an outside system (e.g., a system of one or more processors, one or more computers) that includes CPUs, GPUs, and so on, may perform at least a portion of processing (e.g., generating image information, processing data) and provide information to, and receive information from, local processing and data module 240, remote processing module 250, and / or remote data repository 260, for instance via wireless or wired connections.

[0043] In a display subsystem, there exists a stream-based MIPI interface that transports the data to the endpoint display. This interface usually employs a stream-based compression algorithm that is low latency and predictive based on neighbor pixels, like the DSC compression algorithm. These compression algorithms try to maintain a fixed frame size with fixed compression.

[0044] Embodiments of the present invention utilize an alternating algorithm that implements a complete frame fixed compression, for example, DSC with a simpler, black- only pixel run length encoding algorithm. Therefore, the simpler run length encoding only for "black" executes in some embodiments only when the amount of "black" pixels exceeds a threshold amount in some embodiments, otherwise, the system throttles back to the full frame mechanism.

[0045] FIG. 3 A illustrates a compression-level obtained as a function of time, represented by successive frames, where 100% compression denotes an all black image, according to an embodiment of the present invention.

[0046] According to embodiments of the present invention, the "spatial compression" can include a run length encoding (RLE) having a mask that is prepended or prefixed to a line of video content (i.e., added as a header at the beginning of the data corresponding to a line of video content). The mask includes a series of bits that correspond to groups of pixels of the line. For example, a line with 2048 pixels can be prepended with a mask having 16 bits (i.e., 2 bytes). Each bit represents a group of 128 pixels. The bits of the mask are used to indicate groups of 128 pixels that have zero values (i.e., are black pixels). Thus, the incoming stream of pixels that are processed and delivered to memory and / or the display will be grouped in groups of 128 pixels, skipping over pixel regions that have zero values (i.e., are black pixels).

[0047] Following this example, a mask with values of 16'b 11111111 11111111 can be used to denote a line in which all the pixels have zero value. In this example, no video pixels will be stored in memory or sent from the processor performing the mask definition process to the processor used to generate video content for the display. A mask with values of 16'b00000000_l 1111111 can be used to denote a line in which the first half of the pixels (pixels 0 - 1023) have non-zero values and the second half of the pixels (pixels 1024-2047) have zero value. For this mask, since only the first half of the pixels will be stored in memory, memory and processor requirements are reduced by approximately one half.

[0048] Thus, as the video stream is delivered to the display, the mask is used to identify groups of pixels, i.e., pixel regions, that are set to zero, as well as to provide alignment information for the incoming stream of video pixels that are delivered to memory and / or the display. Since the logic that writes the internal RAM of the display can be aligned to groups of 128 pixels as well, this write logic is able to implement a sequential process that is characterized by reduced power consumption and data bandwidth in which the mask and video data are used to sequentially write to the video data buffer.

[0049] As an example, consider an original frame with a white diamond on a black background. As discussed more fully below in relation to Table 1, the methods and system described herein can be utilized to compress the video image frame and then form a reconstructed image before display to the user. The original frame is analyzed and each line of the frame is represented by an entry in a mask. Only pixels having a brightness greater than a threshold are provided along with the mask, enabling the dark areas of the frame to be skipped when the pixel data is transmitted. As a result, the sparse encoded frame only includes the pixels corresponding to the white diamond shape.

[0050] The embedded pixel compression algorithm effectively adds one extra pixel per line. This added pixel is used as the mask that helps denote which pixel groups (group of linear pixels) include pixels with intensities greater than a threshold (e.g., are not black) and included subsequent to the mask special pixel. This effectively eliminates the need to transmit clean pixel groups (all black pixel groups), thereby only transmitting pixel groups including pixels with intensities greater than the threshold. In addition, in some examples, if the number of pixels having intensities greater than the threshold in a pixel group falls below a threshold number of pixels (e.g., a number of non-black pixels), then the mask can indicate that the pixel group is clean. For example, the threshold number of pixels may be 16. If a pixel group has 128 pixels and only six pixels have an intensity greater than the threshold intensity, then the mask can indicate that the pixel group is clean (all black pixels) since the number of non-black pixels (6) does not exceed the threshold number of pixels (16). Various embodiments of the present invention can use different values for the threshold pixel intensity and the threshold number of pixels in a clean pixel group per frame.

[0051] As a result of the run length encoding, if only 30% of the screen is used for the display of virtual content, then a -70% compression would occur as 70% of the screen is represented by the extra masking pixel corresponding to the lines having all black pixel groups. Considering a white diamond on a black background, many of the initial frames include a significant amount of black pixels. As a result, the compression levels are near 100%.

[0052] FIG. 3B illustrates a histogram of frame count versus compression for a sparsity compression system implementation according to an embodiment of the present invention. As illustrated in FIG. 3B, approximately a majority of the frames achieve a compression of 50%.

[0053] However, as shown in FIG. 3B, a significant number of frames include a limited number of lines with all black pixel groups. For these frames, the mask-based compression method provides a limited amount of data compression as represented by the frames having compression levels less than 50%. Referring to FIG. 3B, a group of frames in section A are characterized by a compression level of -24%, resulting in a significant number of frames with low levels of compression.

[0054] Thus, in order to address the frames that result in a low level of compression using the mask-based compression approach, embodiments of the present invention combine themask-based compression method described above with an algorithm that is used to compress a complete fixed frame, like DSC. As described herein, these methods can be alternated on a frame-by-frame basis, compressing a first frame using the mask-based compression method and the subsequent frame using a frame-based compression method, for example, DSC.

[0055] The DSC algorithm, as an example, has different compression levels, with an impact on image quality. Therefore, if a 24 bit to 15 bit compression setting is used in order to provide a peak signal-to-noise ratio (PSNR) greater than or equal to 60 dB, then this level could be used as a threshold. For the purposes of explanation, this was the level (i.e., 37%) used in the "DSC-sparse" system model herein, however, any level could be used.

[0056] FIG. 4A illustrates a compression-level obtained as a function of time, represented by successive frames versus frequency, for both a sparsity compression system implementation and a DSC-SPARSE system implementation, according to an embodiment of the present invention. In FIG. 4A, each frame was compressed using either the mask-based compression method or DSC in accordance with the alternating algorithm that implements either the mask-based compression method or the complete frame fixed compression, for example, DSC.

[0057] As shown in FIG. 4A, each frame is analyzed and the number of lines having pixels characterized by a brightness level less than a threshold is determined. If the mask-based compression approach will result in a compression level greater than a compression threshold (e.g., 37%), then the frame is compressed using the mask-based compression method. In FIG. 3, this results in the first -3800 frames being compressed using the mask-based compression method.

[0058] If the mask-based compression method will produce a compressed frame with a compression level less than 37%, for example, a frame with very little black content, then the DSC method is utilized. This results in these frames having a 37% compression value. Referring to FIG. 3, the frames represented by the black line, for example the series of frames indicated by ranges A and B, have compression values less than 37% and are compressed using DSC, effectively baselining the minimum compression at 37%. Thus, the frames represented by the black line have a compression value of 37% instead of the lower value that would have been achieved using the mask-based compression method.

[0059] Using some embodiments of the present invention implemented in conjunction with an SRAM display, only a single DSC IP block is utilized. As an example, two channelscould be utilized, a virtual channel for spatial RLE compression, of which spatial RLE (black) is a minimal implementation, and a display channel for DSC compression.Accordingly, lossless spatial compression in the range of 80% can be achieved. As discussed in relation to FIG. 4B, a lower level for the compression can be set, for example, at 37%, 50%, or the like.

[0060] FIG. 4B illustrates a histogram of frame count versus compression for a sparsity compression system implementation and a DSC-SPARSE system implementation according to an embodiment of the present invention. As illustrated in FIG. 4B, the number of frames with compression less than -37% is reduced to zero since either the mask-based compression method was utilized for frames that could be compressed with a compression level greater than 37% or the frame-based compression method (e.g., DSC) was utilized for the remaining frames that could not be compressed with a compression level greater than 37% using the mask-based compression method. Thus, whereas the mask-based compression method operating alone produced a number of frames with a compression level less than 37%, the alternating method provided by embodiments of the present invention limits the lowest compression level to -37% as illustrated in FIG. 4B. For frames with significant black pixel content, the mask-based compression method provides high levels of compression while for frames with limited black pixel content, the frame-based compression method establishes a floor for the compression level, for example, 37% in this illustrated embodiment. As will be evident to one of skill in the art, the minimum compression level does not need to be 37%, which is merely exemplary and other minimum compression levels can be utilized depending on the particular application. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0061] The information on the compression method utilized for each frame can be provided to the endpoint, for example, a decoder or a display in order for the endpoint to utilize the appropriate decompression method when reconstructing each frame.

[0062] FIG. 5 A is a simplified flowchart illustrating a method of displaying a reconstructed image using an alternating compression algorithm according to an embodiment of the present invention. The method 500 includes receiving a frame of video data (510). The method also includes determining a number of lines in the frame having pixel groups characterized by a brightness level less than a threshold (512).

[0063] If the number of lines is less than a threshold (e.g., a compression threshold) (514), then the frame is compressed using a first data compression algorithm (e.g., a frame-based compression method) (520) to form a first compressed image and the first compressed image is stored in memory (522).

[0064] If the number of pixels is greater than or equal to the threshold, then the frame is compressed using a mask-based data compression algorithm to form a second compressed image (524). The second compressed image is formed by forming a mask for each line of the plurality of lines. For each pixel group of the plurality of pixel groups, the mask-based data compression algorithm includes defining a first bit if pixels in the pixel group are characterized by pixel values less than a brightness threshold and defining a second bit if pixels in the pixel group are characterized by pixel values greater than or equal to the brightness threshold. The method also includes providing pixel values for pixels in pixel groups having the second bit and storing the mask and the provided pixel values for each line in a memory (526).

[0065] Frame-based compression systems that can be used to perform a frame-based compression method (520) include DSC, VDC-X, and the like, as well as the methods discussed more fully in relation to FIGS. 8 and 10. Mask-based compression systems that can be used to perform a mask-based compression method (524) are discussed more fully in relation to FIGS. 5B and 5C.

[0066] Referring once again to FIG. 5A, a determination is made as to whether the first data compression algorithm (e.g., a frame-based compression method) or the mask-based compression method was utilized (530). If the first data compression algorithm was used, then the reconstructed image is formed by decompressing the first compressed image (532). If, on the other hand, the mask-based compression method was used, then the reconstructed image is formed by extracting the mask and the provided pixel values for each line from the memory and forming the reconstructed image using the mask and the provided pixel values for each line (534). The reconstructed image is then displayed (536). In some embodiments, the reconstructed image can be transmitted to a display prior to the reconstructed image being displayed.

[0067] If additional frames are present (538), then the method operates on the next frame of video data by receiving a frame of video data (510). Once all of the video frames have been processed and there are no additional video frames, the method ends (539). Accordingly,embodiments of the present invention alternate between compression methods for each frame depending on the level of compression that can be achieved by each compression method.

[0068] It should be appreciated that the specific steps illustrated in FIG. 5 A provide a particular method of displaying a reconstructed image using an alternating compression algorithm according to an embodiment of the present invention. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Moreover, the individual steps illustrated in FIG. 5 A may include multiple sub-steps that may be performed in various sequences as appropriate to the individual step. Furthermore, additional steps may be added or removed depending on the particular applications. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0069] According to embodiments of the present invention, various stream-based compression standards like DSC and VDC-X can be utilized to achieve a low latency implementation. Additionally, the exemplary compression methods discussed in relation to FIGS. 5B and 5C can be utilized.

[0070] FIG. 5B is a simplified schematic diagram illustrating operation of a mask-based compression system according to an embodiment of the present invention. The system 540 receives incoming video 542, for example, virtual content for display on an AR device, at a first frame rate, illustrated as 60 Hz in FIG. 5B. Although video content at 60 Hz is illustrated in this figure, embodiments of the present invention are not limited to this particular frame rate and other frame rates can be utilized in accordance with the present invention. Referring once again to FIG. 2, the incoming video 542, which can be virtual content generated by remote processing module 250, can be received by local processing and data module 240 coupled to display 202.

[0071] The incoming video stream serves as an input to mask definition process 544, which analyzes each video image and generates a mask corresponding to each line of the incoming video stream. An example of a mask is illustrated in relation to Table 1. In an exemplary embodiment, each line of video data is analyzed pixel group by pixel group. Thus, a line of video data can be analyzed to determine whether the pixel values for pixels in the first group of 128 pixels are non-black or alternatively, are greater than a brightness threshold. If pixels in the first group of 128 pixels are black pixels (or pixels with a brightness less than athreshold), then the mask bit corresponding to this pixel group is set to one. The brightness of the pixels in the second group of 128 pixels is then analyzed, and so forth for all of the pixels in the line of video data. This analysis of the pixel groups (and the lines of video data) can be performed sequentially, in parallel, or in combinations thereof. The output of mask definition process 544 will thus be a mask and pixel data for each line of video data.

[0072] The mask and pixel data produced by mask definition process 544 is communicated by a mask and pixel data communication process 546 to memory storage process 548, which can utilize a memory provided as an element of local processing and data module 240 illustrated in FIG. 2. Since pixel data is not stored for pixel groups having a mask bit value equal to one, the mask and pixel data will generally occupy less memory than if all the pixel data was stored.

[0073] When the video content is ready for display, a memory retrieval process 550 is performed to extract the video content from memory and a decompression process 552 using the mask and pixel data is performed. During decompression, pixels in pixel groups having a mask bit value of one are assigned a pixel value corresponding to a black pixel and pixels in pixel groups having a mask bit value of zero are assigned the pixel value stored in memory. Thus, a reconstructed image is formed based on the mask and pixel data. Subsequent processing, including warp and depth correction, can be performed by subsequent processing process 554 prior to display on display 556, which can be display 202 illustrated in FIG. 2. Thus, in this embodiment, the video content is provided to display 556 for display to the user.

[0074] Although a mask size of 16 bits and pixel groups of 128 pixels are used in the context of a display with 2048 pixels per line as an exemplary embodiment, this is merely exemplary and masks with a different size as well as pixel groups of a different size can be utilized in conjunction with displays having a different number of pixels per line. For example, a mask size of 32 bits and pixel groups of 64 pixels could be used to provide higher resolution for systems with appropriate memory and processor capabilities. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0075] Table 1 illustrates the mask values and the pixel data corresponding to a diamondshaped video frame in which a white diamond is present on a black background. As shown in Table 1, the first line of video data has values for the central pixel groups (pixel groups 8 and 9) corresponding to pixels 897-1024 and 1025-1152, whereas other pixel groups have zero value. Thus, the pixel data for pixel groups 1-7 and 10-16 is not communicated during maskand pixel data communication process 546 to memory storage process 548. The second line of video data has values for the four central pixel groups (pixel groups 7, 8, 9, and 10) corresponding to pixels 767-1024 and 1025-1280, whereas other pixel groups have zero value. The central lines of video data (i.e., lines 540 and 541) have values for all pixel groups and, as a result, deliver all pixel data (i.e., pixels 1-2048) during mask and pixel data communication process 546 to memory storage process 548. Similar to lines 1 and 2, lines 1079 and 1080 have values for pixel groups 7, 8, 9, and 10 and pixel groups 8 and 9, respectively. As a result, pixels 767-1024 and 1025-1280 are delivered for line 1079 and pixels 897-1024 and 1025-1152 are delivered for line 1080.Table 1

[0076] Because pixel values are not provided for pixel groups corresponding to bits in the mask that have a value of one, when the data stream is decompressed, the corresponding pixels are black pixels. As an example, if a video line only had black pixels, the mask would be 16' 11111111 11111111 and no pixel data would be delivered. Thus, at the decoding stage, the mask and the pixel values are received and the decoder recreates the line of video data, assigning black pixel values to pixels in pixel groups corresponding to mask values of one and assigning other pixel values according to the received pixel values for mask values of zero.

[0077] It should be noted that, although the discussion has been provided in relation to black pixels that have no pixel values (i.e., (RGB) = (0,0,0)), other embodiments utilize a brightness threshold for the RGB components to encode pixels that are not completely black as black in order to conserve system resources. As an example, if pixels in a pixel group have a maximum RGB bit depth less than 5% of the maximum bit depth, for example, (RGB) < (12,12,12) for a 256 maximum bit depth, then this pixel group can be defined as a "black" pixel group and pixel data for this pixel group will not be communicated. In another exemplary embodiment, a second threshold based on the number of bright pixels in a pixel group can be utilized. As an example, if the number of non-black pixels (i.e., the number of pixels having non-zero pixel values RGB = (>0, or >0, or >0)) is less than a second threshold, then the pixel group can be defined as a "black" pixel group. In this case, a single bright pixel in an otherwise dark field, which can be characteristic of noise, will result in the pixel group being defined as a black pixel group. In some implementations, both thresholds can be used, with only pixel groups having some pixels greater than the brightness threshold, with this number of pixels being greater than the second threshold being defined as non-black pixel groups. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0078] Thus, although this thresholding process will eventually display dark, but not black pixels, as black pixels, the reduction in processing and memory requirements can outweigh the decrease in image quality. Thus, in the mask-based compression process, black pixels as well as dark pixels can be treated as black pixels as appropriate. The threshold can be varied depending on the particular application in view of available processing and memory resources. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0079] As an alternative to the mask illustrated in Table 1 that encodes the image of a white diamond on a black background, rather than defining a mask for one or more lines of data, a prepended value equal to the line number can be utilized. In this alternative embodiment, the default display value for all pixels will be black. For lines that include pixels that are not black, the line number and the pixel data for that line are provided. As an example, if the entire screen was black other than content on lines 540 and 541 (i.e., a colored stripe running across the middle of the screen), the line number (in binary format) and the pixel values could be provided as (line # :: pixel data):0000001000011100 :: pl, p2 ... p2047, p2048; and0000001000011101 :: pl, p2 ... p2047, p2048.

[0080] Thus, in this "line number" embodiment, which can be utilized in place of or in addition to the mask-based method, additional savings in processing and memory overhead can be achieved. In some embodiments, the image frame is analyzed and, if the amount of content defined by black pixels is greater than a threshold, the line number embodiment will be used to compress the image frame, whereas, if the amount of content defined by black pixels is less than a threshold, the mask embodiment will be used to compress the image frame. Thus, headers defining a mask for a line of video data or headers defining the line number of the video data can be utilized according to embodiments of the present invention. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0081] Referring to mask definition process 544, the processor (e.g., an ASIC processor) that transmits data to the display processor analyzes each line of video data and defines the appropriate mask to be prepended to each line of video data. In some embodiments, as discussed in relation to FIG. 5C below, in addition to achieving reduced memory use as a result of the mask and pixel data communication and memory storage processes, display 556 can also incorporate memory reduction techniques and use the mask to decode the incoming stream and display the video image.

[0082] FIG. 5C is a simplified schematic diagram illustrating operation of a mask-based compression system with display decompression 560 according to an embodiment of the present invention. The system illustrated in FIG. 5C shares common elements with the system illustrated in FIG. 5B and the description provided in relation to FIG. 5B is applicable to FIG. 5C as appropriate.

[0083] Referring to FIG. 5C, the processing pipeline starting with the receipt of incoming video 542 through subsequent processing process 554 is identical to that discussed in relation to FIG. 5B. As shown in FIG. 5C, after subsequent processing process 554, the mask and pixel data is communicated to the display by mask and pixel data communication process 562. At the display, the mask and pixel data is stored in memory storage 564. When the video content is ready to be displayed, the mask and pixel data is retrieved from memory at memory retrieval process 566, and the video content is decompressed by decompression process 568 to produce a reconstructed image, and displayed on the display 570.

[0084] Thus, in this embodiment, sparsity compression is performed prior to the mask and pixel data being communicated to the display, thereby reducing communications bandwidth requirements. In these embodiments, the display 570 is operable to implement the decompression process (i.e., the decoding process) to form the reconstructed image prior to display of the video content. As a result, system performance can be improved in cases for which decompression of the mask and pixel data is less resource-intensive than communicating the video content instead of the mask and pixel data. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0085] According to some embodiments of the present invention, there would be an embedded image-line control or alternate control mechanism that, per frame, would provide information to the endpoint display related to which system to use to decode the incoming MIPI frame. In addition, virtual MIPI channels could be utilized to indicate the compression ratio used by the endpoint display.

[0086] Some embodiments of the present invention alter the compression quality based on eye tracking, thus giving the foveated regions a higher compression ratio at a loss of quality. It does this for the MIPI interface, thereby decreasing the amount of data that is sent over MIPI to a liquid crystal on silicon (LCOS) / micro-light emitting diode (pLED) display. Thereby, embodiments also produce a saving in power consumption.

[0087] Embodiments of the present invention reduce the amount of stream-based data sent over MIPI compression that occurs. Moreover, embodiments alter the compression quality based on eye tracking, thus giving the foveated regions a higher compression ratio at a loss of quality. Furthermore, embodiments allow for a higher compression ratio for stream-based compression techniques, and allow for quality to be preserved for the areas being observed bythe user. As a result, embodiments allow for a much higher compression ratio while preserving quality.

[0088] For stream-based compression standards like DSC and VDC-X, a low latency implementation is utilized. This low latency reaction is utilized so that the previous spatial WARP adjustments that are made are still applicable.

[0089] In some embodiments, foveation is performed by having a region of an image with high quality while the rest of the image is at a lower, reduced quality. This foveation process can be combined with the mask-based compression processes and systems described herein to provide additional reductions in processing and memory requirement. Thus, the foveated image compression processes discussed herein can be utilized as the frame-based image compression processes discussed in relation to FIG. 5A. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0090] In some video systems, the incoming video frame rate is lower than the eye-tracking frame rate. For instance, the video frame rate can be 60 Hz, but the eye tracking system can generate eye gaze information at 120 Hz. In the context of these systems, embodiments of the present invention provide a dynamically varying window size for the foveated region. In some embodiments, the window size is directly proportional to the current speed of the eye movement. Thus, embodiments of the present invention are able to maintain a reduced, un- foveated window size (e.g., the smallest possible un-foveated window size). Receiving incoming video frames at a first rate and eye tracking information at a higher, second rate, the system described herein dynamically sizes the foveation window based on the eye gaze location and / or velocity to reduce the foveation window size and data processing and memory resource utilization as a result.

[0091] It should be noted that, by utilizing embodiments of the present invention, not only can the foveation window size be varied dynamically, but the eye tracking sample rate can be varied as well. For example, the size of the window could be increased and the eye tracking sampling rate could be decreased to 60 Hz. As a result, embodiments of the present invention provide benefits over conventional systems since eye tracking consumes power, which can be reduced by the dynamic variation of the foveation window size and / or the eye tracking sampling rate.

[0092] Some embodiments of the present invention utilize dynamic foveation based on user eye gaze to decrease memory access and data transmission requirements. In particular,embodiments provide eye gaze information to a foveation process in order to define a foveation aperture based on eye gaze position and / or velocity. Therefore, embodiments of the present invention are able to utilize individual compression quality settings for different portions of an image, which provides benefits not available using methods in which the whole image has a single compression quality setting.

[0093] Embodiments of the present invention are explained in relation to foveation of images, but are applicable to a variety of encoding standards, including JPEG and MPEG compression standards and / or sub-sampling of the image. In particular, embodiments of the present invention are applicable to image and video compression operations in which the quality setting is variable across the image. As described more fully herein, utilizing the methods and systems discussed herein, different portions of an image can be selected based on the eye gaze and subsequently compressed using different quality settings, with portions of the image adjacent the location of the eye gaze being compressed with a higher quality setting and portions of the image more distant from the location of the eye gaze being compressed with a lower quality setting, thereby enabling reductions in the amount of data that is stored, transmitted, and the like. Since the user is looking at the eye gaze location, the more lossy compression utilized with portions of the image more distant from the eye gaze location has a reduced impact on user experience while reducing processing and memory requirements.

[0094] In conventional systems, MPEG compression is implemented at a fixed quality that does not take into account the human gaze. By knowing where the human gaze is currently located and taking the human gaze into account, embodiments of the present invention can reduce the quality (i.e., the bandwidth) at locations in an image where the user is not looking, i.e., locations in the image that are spatially separated from the eye gaze location, thereby decreasing the image quality in these regions and decreasing the overall need to send something at a superior quality setting that the human eye would not be able to discern, because the human eye is not currently focused on these non-gaze locations. Thus, embodiments of the present invention provide a video compression algorithm that takes human gaze into account and creates a foveated compression algorithm dependent on human gaze.

[0095] FIG. 6 is a plot of PSNR (dB) as a function of compression level for several use cases according to an embodiment of the present invention. The PSNR, measured indecibels, is a ratio of the peak signal-to-noise between two images. This ratio can be used as a quality measurement between the original image and the compressed image.

[0096] For the use cases 610-620 (i.e., the six images) illustrated in FIG. 6, different videos with different types of virtual content, including natural images such as a flower, computer generated images, text, and the like were utilized and compressed using different levels of compression. Five columns are utilized to illustrate five levels of compression: 24 — > 15 bits per pixel (bpp), 24 — > 12 bpp, 24 — > 10 bpp, 24 — > 8 bpp, and 24 — > 6 bpp, representing compression from a 24 bit color encoding to 15 bits, 12 bits, 10 bits, 8 bits, and 6 bits, respectively. The amount of compression achieved is also illustrated: 37%, 50%, 58%, 67%, and 74%, respectively. The PSNR is shown for each video use case at each level of compression, ranging from ~85 dB to ~50 dB for 24 bit to 15 bit compression to ~57 dB to ~30 dB for 24 bit to 6 bit compression.

[0097] In some applications, it is desirable to maintain a PSNR greater than 60 dB in order to provide a desired user experience. In particular, text that is compressed and characterized by a PSNR less than 60 dB has been determined to result in an undesirable user experience. Given this 60 dB threshold, the inventors have determined that most of the use cases illustrated in FIG. 6 have a PSNR greater than 60 dB at a compression level of 37% (i.e., 24 — 15 bpp compression level). For example, use case 610 (Image 1) and use case 612 (Image2) have PSNR greater than 80 dB at a compression level of 37%, while used case 614 (Image3) and use case 616 (Image 4) have PSNR greater than 60 dB at a compression level of 37%. In contrast, at a higher compression level of 50% (i.e., 24 —> 12 bpp compression level), approximately half of the use cases (e.g., use cases616 (Image 4), use case 618 (Image 5), use case 620 (Image 6) are characterized by a PSNR less than 60 dB. Thus, some embodiments of the present invention utilize a compression level of 37% to compress one or more portions of an image as described more fully below in relation to FIG. 6.

[0098] FIG. 7 is a simplified image illustrating an image frame divided into a primary quality region and a secondary quality region according to an embodiment of the present invention. The image 700 illustrated in FIG. 7 includes a primary quality region 710, also referred to as a high quality region, and a secondary quality region 720, also referred to as a low quality region. As discussed more fully below, the primary quality region 710 will be compressed and decompressed using a first quality setting or compression level and the secondary quality region 720, or the entire image, will be compressed and decompressedusing a second quality setting or compression level characterized by a lower quality setting or a lower compression level than that used for the primary quality region, thereby providing memory savings and other benefits.

[0099] DSC

[0100] Conventional DSC does not provide for variable quality compression of different portions of an image frame. Rather, DSC takes a 24 bit color encoding for an image frame and compresses it down to 15 / 12 / 10 / 8 bits. The higher the compression (24 —> 8 bpp), the worse the impact to quality. As to the quality utilized for the section that the eye is focused upon, embodiments are able to maintain, for example, a PSNR quality setting above 60dB as discussed above. From the use case analysis illustrated in FIG. 6, the inventors have determined that this only occurs at a 37% compression configuration (24 — 15 bpp). However, only the area in which the eye is currently focused on actually utilizes that compression setting. The outer foveated region (e.g., the portion of the image more distant from the eye gaze location) can afford to have a lower quality, for example, 75% compression level (24 —> 8 bpp).

[0101] Therefore, for a neighbor-based compression standard like DSC, where there is no concept of tiles, embodiments divide the main screen into a primary quality region, also referred to as a high quality region, and secondary quality region, also referred to as a low quality region (as shown in FIG. 7) or smaller sections (as shown in FIG. 9) each with a different compression ratio. The selected compression ratio will be a function of the current eye gaze location. Thus, referring to FIG. 7, in which the eye gaze location is positioned inside the primary quality region 710, the primary quality region 710 can be compressed with a lower compression level (e.g., 24 — > 15 bpp), and the secondary quality region 720 can be compressed with a higher compression level (e.g., 24 — > 8 bpp). In conjunction with each frame, the location and size of the primary quality region can be communicated and utilized during reconstruction of the image. In some examples, the secondary quality region 720 can be compressed with an even higher compression level (e.g., 24 — > 6 bpp) in order to reduce memory and processing utilized by the system. In embodiments in which the entire image is compressed using the higher compression level (i.e., the lower quality level) as described more fully herein, the primary quality region 710 can be overlaid on the entire image when the image is reconstructed. As an example, for a 2K x 2K image (i.e., a 4M pixel image) witha IK x IK primary quality region corresponding to the eye gaze location, a memory reduction on the order of 50% can be achieved:2K x 2K * (1 - 0.7) = 1.258 MpixIK x IK * (1 - 0.37) = 0.661 MpixTotal = 1.92 Mpix

[0102] Thus, embodiments of the present invention can provide a reduced bandwidth for image communication of -50% of the original 2K x 2K image bandwidth. Benefits of embodiments of the present invention include a constant bandwidth reduction (e.g., -50%) while only adversely impacting resolution in portions of the image distant from the eye gaze location. Using a SRAM display, two DSC decoders can be utilized with two streams for rasterization to occur with reduced latency: a virtual channel can be used as a second channel; and the channel can denote the compressing setting used. In some implementations, the display ASIC can perform a merge overlay to reconstruct the compressed image frame.

[0103] FIG. 8 is a simplified flowchart illustrating a method 800 of compressing an image using different compression ratios for a primary quality region and a secondary quality region, according to an embodiment of the present invention. The method 800 includes receiving an image (810) and determining an eye gaze location of a user (812). The eye gaze location can be determined using an eye tracking camera of an augmented reality device. Moreover, the image can be virtual content generated by the augmented reality device.

[0104] The method also includes compressing a primary quality region using a first compression ratio (e.g., X — > Y bpp) to form a primary quality image (814) and compressing a secondary quality region using a second compression ratio (e.g., X — > Z bpp, where Z < Y) to form a secondary quality image (816). Generally, the primary quality section includes the eye gaze location where high quality compression is desired. For example, the primary quality section can be a rectangular shape centered at the eye gaze location as illustrated in FIG. 7. Thus, the image can be an M x N image and the primary quality region comprises an aM x aN image, where a<l

[0105] As examples of various implementations, the first compression ratio and the second compression ratio can utilize compression values such as X = 24, Y = 15 and Z = 12; X = 24, Y = 15 and Z = 10; X = 24, Y = 15 and Z = 8; or X = 24, Y = 15 and Z = 6. These values aremerely exemplary and other values for the compression ratios can be utilized depending on the particular application.

[0106] In some embodiments, a foveation map is generated to define the primary quality region and / or the secondary quality region based on the eye gaze location. As an example, the secondary quality region of the image can consist of the image, resulting in the entirety of the image being compressed at the second compression ratio. In these embodiments, as discussed below, overlaying the primary quality image on the secondary quality image will include overlaying the primary quality image compressed with a low level of compression on a version of the image compressed using a higher level of compression.

[0107] In order to form a reconstructed image, the method further includes overlaying the primary quality image on the secondary quality image to form the reconstructed image (818) and displaying the reconstructed image (820). In some embodiments, the method can also include post-processing image content in at least one of the primary quality region or the secondary quality region or performing image processing on the reconstructed image prior to displaying the reconstructed image.

[0108] The image may be an image included in a video stream. Determining the eye gaze location of the user can utilize an eye tracking system that provide the eye gaze location as a function of time. The foveation map defines the compression ratio with which portions of the image are compressed and varies as a function of position in the image with respect to the eye gaze location, with region(s) close to the eye gaze location being compressed using a lower compression ratio and region(s) more distant from the eye gaze location being compressed using a higher compression ratio. In the example illustrated in FIG. 7, two regions are included in the foveation map, but the present invention is not limited to this particular implementation and three regions or more than three regions can be defined. In some embodiments, the foveation map includes a first region of the image and a second region of the image. The method 800 may be referred to as an N-way compression (e.g., DSC, VDC- X, or JPEG), where N refers to the number of regions determined for the image. For example, based on the eye gaze location, a primary quality region, an intermediate quality region, also referred to as a medium quality region, surrounding the primary quality region, and a secondary quality region can be determined for the image. The techniques of method 800 can then be used as a 3 -way compression, with different compression ratios for each region.

[0109] Referring back to FIG. 7, in some examples the secondary quality region 720 can encompass the entire image, including the portion of the image in the primary quality region 710 characterized by the eye gaze location. When decoding the compressed image (e.g., for reconstruction for display to a user), it may desirable to decode the sections of the image in parallel. For an image divided into a primary quality region 710 and a secondary quality region 720 as in FIG. 7, the secondary quality region 720 may be considered as the entire image. For example, for a 2 kilopixel x 2 kilopixel image (4 megapixel total), the secondary quality region 720 may be the entire 4 megapixel image and may be compressed using a high compression level (e.g., 24 — > 8 bpp). The primary quality region 710 may be determined based on the current eye gaze location and may be, for example, a 1 kilopixel by 1 kilopixel region (1 megapixel total). The primary quality region 710 can be compressed using a low compression level (e.g., 24 — > 15 bpp). Therefore, two DSC decoders can be used to decode the compressed image. During reconstruction of the image, the decoded primary quality region can be overlaid on the decoded secondary quality region.

[0110] FIG. 9 is a simplified image illustrating an image frame divided into primary quality sections and secondary quality sections according to an embodiment of the present invention. As discussed more fully below, the sectioned image frame 900 illustrated in FIG. 9 can be utilized to define a foveation map that defines the compression ratio with which different sections of the image are compressed in such a manner that the compression ratio or other compression quality metric varies as a function of position in the image with respect to the eye gaze location. As an example, sections close to the eye gaze location can be compressed using a lower compression ratio and sections that are more distant from the eye gaze location can be compressed using a higher compression ratio.

[0111] Referring to FIG. 9, the four sections 910, 912, 914, and 916 including the primary quality region 902 (i.e., the region corresponding to the current eye gaze location) will be compressed with a lower compression level (e.g., 24 — > 15 bpp) and the remaining sections, which can be referred to as peripheral sections or secondary quality sections, will be compressed with a higher compression level (e.g., 24 — > 8 bpp). The compression settings for the various sections of each image frame can be communicated and used during reconstruction. As an example, for a binary setting (high / low), only one bit per section can be utilized to denote the compressing settings, utilizing 16 bits in the example illustrated in FIG. 9. As a result, when the compressed image is reconstructed for display to the user, the primary quality region, which corresponds to the eye gaze location, is characterized by higherquality than the remainder of the image, which is more distant from the eye gaze location. As a result, embodiments of the present invention provide a foveated image based on the eye gaze location with reduced storage and transmission requirements.

[0112] As an example, for a 2K x 2K image (i.e., a 4M pixel image) with a IK x IK primary quality region corresponding to the eye gaze location and four sections, a memory reduction on the order of 60% can be achieved:12 * (512 x 512) * (1 - 0.7) = 0.944 MpixIK x IK * (1 - 0.37) = 0.661 MpixTotal = 1.604 Mpix

[0113] Thus, embodiments of the present invention can provide a reduced bandwidth for image communication of -60% of the original 2K x 2K image bandwidth. Benefits of embodiments of the present invention include a constant bandwidth reduction (e.g., -60%) while only adversely impacting resolution in portions of the image distant from the eye gaze location. Using a SRAM display, four DSC decoders can be utilized with four streams, one for each of the four parallel DSC blocks for rasterization can occur with reduced latency.

[0114] In some embodiments of the example illustrated in FIG. 9, all sections 910-946 of the image may be compressed at the high compression ratio (e.g., 24 — > 8 bpp). The four sections 910, 912, 914, and 916 including the primary quality region can also be compressed with a lower compression ratio (e.g., 24 — > 15 bpp). Using decoders, all sections 910-946 compressed with the high compression ratio can be decoded according to the higher compression ratio, and the four sections 910, 912, 914, and 916 compressed with the lower compression ratio can be decoded according to the lower compression ratio. During reconstruction of the image, the decoded primary quality sections 910, 912, 914, and 916 can be overlaid on the decoded secondary quality sections 910-946. In some embodiments, the foveation map may define sections that are coincident with the primary quality region. For example, sections 910-916 may include only the primary quality region characterized by the eye gaze location, without including portions of the image in the secondary quality regions.

[0115] As with the N-way compression, it may be desirable to use multiple DSC decoders to decode the compressed image in the section-based DSC technique. For example, four DSC decoders can be used to decode the compressed image, with one decoder used to decode the primary quality sections 910-916, another decoder used to decode the sections 920-926, athird decoder used to decode the sections 930-936, and a fourth decoder used to decode the sections 940-946, with each decoder using a compression ratio for each group of sections based on proximity to the eye gaze location. In some embodiments, depending on the memory capacity (e.g., SRAM) of the system used to decode, a single decoder may be implemented with acceptable latency when decoding the compressed image.

[0116] The image may be an image included in a video stream. Determining the eye gaze location of the user can utilize an eye tracking system that provide the eye gaze location as a function of time. The foveation map defines the compression ratio with which different sections (e.g., sections 910-916, sections 920-926, sections 930-936, and sections 940-946) of the image are compressed and varies as a function of position in the image with respect to the eye gaze location, with sections close to the eye gaze location being compressed using a lower compression ratio and sections more distant from the eye gaze location being compressed using a higher compression ratio. In the example illustrated in FIG. 9, 16 sections are included in the foveation map, but the present invention is not limited to this particular implementation and more or fewer than 16 sections can be defined. The methods described herein may be referred to as section-based compression (e.g., DSC, VDC-X, or JPEG) methods.

[0117] Although only two compression levels are illustrated in some of the above examples, embodiments of the present invention are not limited to these particular compression levels, but additional number of levels of compression can be utilized. For example, sections 910-914 could be compressed using a 37% compression level (i.e., 24 —> 15 bpp) while sections 920, 922, 924, and 926, which are more distant from the primary quality region, could be compressed using a 50% compression level (i.e., 24 — > 15 bpp), sections 930, 932, 934, and 936, which are more distant from the primary quality region than sections 920-926, could be compressed using a 58% compression level (i.e., 24 —> 12 bpp), and sections 940, 942, 944, and 946, which are most distant from the high quality region than sections 910-916, could be compressed using a 67% compression level (i.e., 24 — > 8 bpp). Thus, the use of two compression levels is merely exemplary. Furthermore, for some sections, the compression level may be 0%, i.e., uncompressed, including sections corresponding to the eye gaze location and primary quality region. Thus, the compressed image could have uncompressed sections as well as compressed sections. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0118] Furthermore, although only sixteen uniform area sections are illustrated in FIGS. 9, this is not required and other numbers of sections, including sections with differing sizes can be utilized, with smaller sections adjacent the primary quality region and larger sections, for example, sections compressed at higher levels, at greater distances from the primary quality region. Thus, the number of compression levels, the levels of compression, the number of the sections, and the sizes of the sections can be varied as appropriate to the particular application. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0119] As the frame size is decreased as a result of the compression of the image, the communication interface, e.g., the MIPI interface, can be modified to enter a low-power data transmission mode or even enter an ultra-low-power sleep mode, thereby saving compute resources and reducing power consumption. At the end point, reconstruction of the compressed image can be performed prior to display to the user.

[0120] FIG. 10 is a simplified flowchart illustrating a method 1000 of displaying a reconstructed image using different compression ratios for primary quality sections and secondary quality sections, according to an embodiment of the present invention. The method discussed in relation to FIG. 10 shares common elements with the method discussed in relation to FIG. 8 and the discussion provided in relation to FIG. 8 is applicable to the discussion provided below in relation to FIG. 10 as appropriate.

[0121] The method 1000 includes receiving an image (1010), dividing the image into N sections (1012), and determining an eye gaze location of a user (1014). As an example, as illustrated in FIG. 9, N =16, resulting in the image being divided into 16 equal pixel size subimages. The image can be virtual content generated by an augmented reality device and the eye gaze location can be determined use of an eye tracking camera of the augmented reality device. In the example illustrated in FIG. 9, the image comprises an M x N image and the primary quality region comprises an aM x aN image, where a<l.

[0122] The method also includes compressing N-M sections (e.g., compressing 4 sections of the 16 sections making up the image when N = 16 and M = 12) using a first compression ratio (e.g., X — > Y bpp) to form a primary quality image (1016) and compressing M sections (e.g., compressing 12 sections of the 16 sections making up the image when N = 16 and M = 12) using a second compression ratio higher than the first compression ratio (e.g., X — > Z bpp, wherein Z < Y) to form a secondary quality image (1018). Generally, the primaryquality section corresponds to the eye gaze location and the primary quality section can have a rectangular shape centered at the eye gaze location. As an example, compressing the N-M sections using the compression ratio X — > Y bpp can include compressing all blocks in the N- M sections of the image using the same compression ratio of X — > Y bpp. Moreover, compressing the M sections using the compression ratio X — > Z bpp can include compressing all blocks in the M sections of the image using the same compression ratio of X — > Z bpp.

[0123] As examples of various implementations, the first compression ratio and the second compression ratio can utilize compression values such as X = 24, Y = 15 and Z = 12; X = 24, Y = 15 and Z = 10; X = 24, Y = 15 and Z = 8; or X = 24, Y = 15 and Z = 6. These values are merely exemplary and other values for the compression ratios can be utilized depending on the particular application.

[0124] In some embodiments, the method also includes generating a foveation map including the N-M sections (i.e., the primary sections corresponding to the eye gaze location) and the M remaining sections (i.e., the secondary sections corresponding to portions of the image more distant from the eye gaze location).

[0125] The method also includes overlaying the primary quality image on the secondary quality image to form the reconstructed image (1020) and displaying the reconstructed image (1022). In some embodiments, the method can include post-processing image content in at least one of the primary quality region or the secondary quality region or performing image processing on the reconstructed image prior to displaying the reconstructed image.

[0126] It should be appreciated that the specific steps illustrated in FIGS. 8 and 10 provide particular methods of displaying reconstructed images according to an embodiment of the present invention. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Moreover, the individual steps illustrated in FIGS. 8 and 10 may include multiple sub-steps that may be performed in various sequences as appropriate to the individual step. Furthermore, additional steps may be added or removed depending on the particular applications. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0127] VDC-X

[0128] The VDC-X compression standard (e.g., VDC-M) uses a tile-based approach instead of a nearest neighbor approach. This compression standard encodes different tiles at different quality settings, however the goal of this conventional compression is to maintain an overall constant frame size (i.e., bit rate). So once a compression ratio is selected it varies each tile in order to maintain the constant bit rate. Using this compression standard in conjunction with embodiments of the present invention, video images are compressed, not solely based on bit rate, but based on the user's eye gaze location. As an example, the four sections 910, 912, 914, and 916 including the primary quality region (i.e., the region corresponding to the current eye gaze location) will be compressed with a higher quality setting than the remaining sections, which can be referred to as peripheral sections, which will be compressed with a lower quality setting that that used for the sections 910-916.

[0129] Some embodiments of the present invention do not maintain a constant bit rate, so that each frame size varies over time, and that the transport interface, for example, MIPI, is put into a low power mode when not in use.

[0130] In a manner similar to the DSC-based approach discussed above, for a VDC-X tilebased approach, embodiments encode the quality of each tile based on the current location of the user's eye-gaze. As illustrated in FIG. 9, using the eye gaze information provided by the eye gaze tracking system of the AR system, tiles are compressed using the VDC-X standard as a function of the distance of the tile from the eye gaze location.

[0131] Therefore, embodiments of the present invention are able to vary the frame size or bit rate per frame, and to use the current eye-gaze information in order to select which tile (VDC-X) or section (DSC) has a higher quality vs the foveated regions that have a lower quality setting.

[0132] In some embodiments, the N-way compression or the section-based compression described above can implement JPEG as the compression standard rather than DSC or VDC- X. In these embodiments, the compression ratios used for the high quality / low quality regions and / or the high quality / low quality sections can instead refer to the quality settings of the JPEG standard.

[0133] Embodiments of the present invention maintain lossless or less-lossy compression on the sections of the image that the eye is focused on, while increasing the compression on the sections of the image that the eye is not focused on. These different compression settings are stored in a foveation map. Therefore, a foveation map can be passed to the compressionengine. In turn, the compression engine can selectively alter predetermined video blocks corresponding to the eye gaze location in order to compress these predetermined video blocks with high quality, while other blocks can be compressed with low quality.

[0134] The foveation map can be created based on eye gaze information, namely, by being able to actively tell where the human eye is currently focused or looking. In embodiments of the present invention, the foveation map is supplied to the encoder and passed to the decoder.

[0135] As discussed below with reference to FIGS. 11-17, foveation maps and foveated images based on eye gaze information are described with respect to JPEG compression and related quality settings. However, the details of determining regions of a foveation map apply to embodiments described above using DSC and VDC-X techniques. The terms "quality" and "quality setting" in the sense of JPEG compression can refer to the compression level (e.g., 24 — > 15 bpp) discussed above with respect to the DSC and VDC-X techniques.

[0136] FIG. 11 is a line drawing illustrating an image compressed using a single quality setting. In this case, all of the pixels in the image are compressed using a conventional process that utilizes a single quality setting for the pixels. Although this process achieves uniform image compression across the image, the inventors have determined that processing and memory requirements can be reduced if portions of the image that are distant from the location where the user is looking are compressed with reduced quality compared to the portion of the image corresponding to the location where the user is looking, while still achieving a desired user experience.

[0137] FIG. 12 is a line drawing illustrating a foveated image with three foveated regions according to an embodiment of the present invention. The image in FIG. 12 is divided into multiple regions based on the eye gaze location. In this case, the user is gazing at the center of the image resulting in the eye gaze location being located at the center of the image. As discussed herein, the eye gaze location can be determined using an eye tracking system as discussed in relation to FIG. 18. Accordingly, the image can be divided into a central region corresponding to the eye gaze location and peripheral regions that are more distant from the eye gaze location. In some embodiments, a foveation map is created based on the eye gaze location, with portions of the image close to the eye gaze location mapping to high quality settings and portions of the image more distant from the eye gaze location mapping to lower quality settings. In FIG. 12, the foveation map takes the form of two peripheral regions with a lower quality setting and a central region with a higher (e.g., 100%) quality setting.

[0138] In the image illustrated in FIG. 12, region 1210, corresponding to the left quarter of the image (i.e., the left 1 / 4), has been compressed using a first quality setting. Additionally, region 1230, corresponding to the right quarter of the image (i.e., the right 1 / 4), has been compressed using the first quality setting. However, region 1220, corresponding to the middle half of the image (i.e., the center 2 / 4), has been compressed using a second quality setting higher than the first quality setting. This division of the image into portions can be referred to as a tri-region division: left quarter (e.g., foveated at 70% quality setting), center half (e.g., un-foveated at 100% quality setting), and right quarter (e.g., foveated at 70% quality setting).

[0139] Although FIG. 12 illustrates division into three regions with a foveation map including these three regions, the present invention is not limited to this implementation and the image can be divided in other manners. By dividing the image into multiple regions, the quality setting for individual blocks or tiles (e.g., 8 x 8 pixel blocks for JPEG compression) included in each region can be set at a predetermined quality setting for each block. Thus, in FIG. 9, all of the blocks in each region are assigned the same quality setting, i.e., the blocks in section 910 are assigned a first quality setting (e.g., 70%), the blocks in section 920 are assigned a second quality setting (e.g., 100%), and the blocks in section 930 are assigned the first quality setting (e.g., 70%), but this is not required and the individual blocks in a region can be assigned different quality settings. Thus, the foveation map can be more complex than the three region division illustrated in FIG. 11. In some embodiments, a foveation map in which blocks in the peripheral regions are assigned quality settings that depend on the distance of the block from the eye gaze location, while blocks in the central region have a uniform quality setting. In other embodiments, the foveation map can be defined such that blocks in the peripheral regions are assigned a uniform quality setting, while blocks in the central region are assigned quality settings that depend on the distance of the block from the eye gaze location. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0140] In the tri-region foveated image illustrated in FIG. 12, a -67% overall reduction of image / memory size was achieved, while retaining 100% quality in region 1220, i.e., the un- foveated section. As discussed above, the region that is unfoveated (i.e., uncompressed or compressed using a lossless compression algorithm) can be any region as identified in the foveation map. As a result, the tri-region division illustrated in FIG. 12 is merely exemplary.

[0141] It should be noted that if the eye gaze location was, for example, on the right side of the image, the foveation map could compress the right side using a higher quality setting and the left side of the image using a lower quality setting. Thus, in this example, if the eye gaze location was within region 1230, region 1210 and region 1220 would be compressed using a first quality setting and region 1230 would be compressed using a second quality setting higher than the first quality setting. As a result, the foveation of the image is a function of the eye gaze location, compressing or encoding the region including the eye gaze location with a higher quality setting than one or more regions more distant from the eye gaze location. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0142] Moreover, although a set of vertical regions is illustrated in FIG. 12, this is not required by embodiments of the present invention and the definition of the regions can be performed in other manners, including horizontally oriented regions, regions defined based on distance to the eye gaze location, for example, a radially-defined set of regions, or the like.

[0143] FIG. 13 is a second foveated image with post-processing in the foveated regions according to another embodiment of the present invention. After post-processing of the image illustrated in FIG. 12, the blurring of the image content in the foveated regions, i.e., region 1210 and region 1230, reduces the artifacts present in these regions.

[0144] FIG. 14 is a foveated 3D generated image with three foveated regions according to yet another embodiment of the present invention. In FIG. 14, the regions are defined in a manner similar to that illustrated in FIGS. 12 and 13. However, the compression can be much higher since, for the 3D generated image, large portions of the image are black. Using the methods described herein, 87% compression was achieved while maintaining 100% quality in the center of the image corresponding to the eye gaze location. In this example, region 1420 was compressed using a 100% quality setting (un-foveated at 100% quality setting) while region 1410 and region 1430 were compressed at lower quality settings. Since, for many instances of virtual content, the image content is highest near the eye gaze location and peripheral regions are dark or black, embodiments of the present invention are particularly well suited for use with virtual reality and augmented reality implementations.

[0145] FIG. 15 is a line drawing illustrating an image that can be utilized in conjunction with multiple foveation maps according to an embodiment of the present invention. In FIG. 15, an image is represented that includes a person 1506 located in section 1510, a tree 1502 located in sections 1520, 1522, 1530, 1532, and a house 1504 located in sections 1524-1526,1538, and 1540. Depending on the eye gaze location, different foveation maps can be created based on this image.

[0146] If the user's eye gaze location is in one of sections 1520, 1522, 1530, or 1532, i.e., the user is looking at the tree 1502, then a foveation map can be utilized in which the blocks in sections 1520, 1522, 1530, and 1532 are compressed using a 100% quality setting (un- foveated at 100% quality setting) while the blocks in the remaining sections (i.e., sections 1510, 1512, 1514, 1516, 1524, 1526, 1528, 1534, 1536, 1538, 1540, and 1542) are compressed using a lower quality settings. Accordingly, compression of the image can be implemented using a foveation map that maintains the quality in the region of the image corresponding to the eye gaze location and peripheral portions of the image can be compressed using a lower quality setting to save system resources including memory and processing.

[0147] Alternatively, if the user's eye gaze location is in one of sections 1524, 1526, 1538, or 1540, i.e., the user is looking at the house 1504, then a foveation map can be utilized in which the blocks in sections 1524, 1526, 1538, and 1540 are compressed using a 100% quality setting (un-foveated at 100% quality setting) while the blocks in the remaining sections (i.e., sections 1510, 1512, 1514, 1516, 1520, 1522, 1528, 1530, 1532, 1534, 1536, and 1542) are compressed using a lower quality settings. Accordingly, compression of the image can be implemented using a foveation map that maintains the quality in the region of the image corresponding to the eye gaze location and peripheral portions of the image can be compressed using a lower quality setting to save system resources including memory and processing.

[0148] Finally, if the user's eye gaze location is in section 1510, i.e., the user is looking at the person 1506, then a foveation map can be utilized in which the blocks in section 1510 are compressed using a 100% quality setting (un-foveated at 100% quality setting) while the blocks in the remaining sections (i.e., sections 1512, 1514, 1516, 1520, 1522, 1524, 1526, 1528, 1530, 1532, 1534, and 1536, 1538, 1540, and 1542) are compressed using a lower quality settings. In some embodiments, the quality settings used for the remaining sections are varied, for example, as a function of distance from the eye gaze location. In these embodiments, blocks in sections 1512, 1514, and 1516 could be compressed using a quality setting of 90%, blocks in sections 1520, 1522, 1524, 1526, and 1528 could be compressed using a quality setting of 80%, and blocks in sections 1530, 1532, 1534, 1536, 1538, 1540,and 1542 could be compressed using a quality setting of 70%. Accordingly, compression of the image can be implemented using a foveation map that maintains the quality in the region of the image corresponding to the eye gaze location and peripheral portions of the image can be compressed using a lower quality setting to save system resources including memory and processing.

[0149] FIG. 16 is a simplified flowchart illustrating a method of compressing an image according to an embodiment of the present invention. The method 1600 includes receiving an image (1610), determining an eye gaze location of a user (1612), and generating a foveation map based on the eye gaze location (1614).

[0150] The image may be an image included in a video stream. Determining the eye gaze location of the user can utilize an eye tracking system that provides the eye gaze location as a function of time. The foveation map defines the quality with which blocks are compressed and varies as a function of position in the image, with blocks in region(s) close to the eye gaze location being compressed using a higher quality setting and blocks in region(s) more distant from the eye gaze location being compressed using a lower quality setting. In the example illustrated in FIG. 9, three regions are included in the foveation map, but the present invention is not limited to this particular implementation and two regions or more than three regions can be defined. Moreover, the blocks in a given region can be compressed using a uniform quality setting or can be compressed with different quality settings, depending on the particular implementation. In some embodiments, the foveation map includes a first region of the image and a second region of the image.

[0151] The method also includes compressing the first region of the image using a first quality setting and the second region of the image using a second quality setting (1616). In some embodiments, the first quality setting is an uncompressed quality setting or lossless compression quality setting. Thus, the blocks in the first region are compressed with higher quality than other portions of the image. The second quality setting is a lower quality setting, for example, a 70% quality setting that reduces the data corresponding to the compressed image in these regions. As discussed above, since the user's eye gaze results in these regions being in the peripheral vision of the user, the loss in quality is offset by the savings in memory and processor usage. The data compression processes for the first region and the second region can be performed sequentially or in parallel, depending on the particular application.

[0152] The compressed image or video, which can be referred to as a foveated image or video, can be transmitted to a display system, along with the foveation map (1618), or can be stored in memory, along with the foveation map (1619).

[0153] In embodiments in which the compressed image or video, along with the foveation map, is stored in memory, the method 1600 includes retrieving the foveated image and the foveation map from memory (1620) and decompressing the first region of the image using the first quality setting and the second region of the image using the second quality setting (1640). In embodiments in which the compressed image or video, along with the foveation map, is transmitted to a display system, the method 1600 includes receiving the foveated image and the foveation map from memory (1620) and decompressing the first region of the image using the first quality setting and the second region of the image using the second quality setting (1640). The decompression processes for the first region and the second region can be performed sequentially or in parallel, depending on the particular application. The two regions can be merged to form the final image suitable for display (1642). The final image is then displayed on the display device (1644).

[0154] It should be appreciated that the specific steps illustrated in FIG. 16 provide a particular method of compressing an image according to an embodiment of the present invention. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Moreover, the individual steps illustrated in FIG. 16 may include multiple sub-steps that may be performed in various sequences as appropriate to the individual step. Furthermore, additional steps may be added or removed depending on the particular applications. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0155] FIG. 17 is a simplified schematic diagram illustrating a gaze-based image foveation system according to an embodiment of the present invention. Referring to FIG. 17, the gazebased image foveation system 1700 includes a wearable 1710 (e.g., a wearable including an ASIC that performs the illustrated operations) that receives an image or a video suitable for display to a user. The image or video can be received using one or more communication interfaces 1720. In the illustrated embodiment, WiFi, USB, DisplayPort (DP), or other communication protocols are utilized to receive the image or video content. In this embodiment, the uncompressed content is MPEG video.

[0156] The wearable 1710 also receives eye gaze information from an eye tracking system 1705. The eye tracking system 1705 can include one or more sensors suitable for measuring eye position and orientation and can provide data that can be utilized by eye gaze processor 1730 in calculating the user's eye gaze. In the embodiment illustrated in FIG. 17, the eye gaze processor 1730 is implemented using a CPU or NPU controller, although other processors can be utilized. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0157] As shown in FIG. 17, the image or video is passed to image compression processor 1722 in some embodiments, which implements a process to form a compressed image / video (e.g., a foveated image / video) based on the user's eye gaze as discussed more fully herein. Different foveation processes can be utilized as appropriate to the particular application, including tile-based foveation processes, sparsity -based compression processes, or the like. In some embodiments, image compression processor 1722 is bypassed and the image or video is passed to memory 1724 for storage.

[0158] When the image or video, is either compressed or uncompressed, depending on the use of image compression processor 1722, an image decompression process can be performed using decompression processor 1726 and the eye gaze information provided by eye gaze processor 1730. In embodiments in which image compression processor 1722 was bypassed and the image or video was passed to memory 1724 for storage, the decompression processor 1726 can be bypassed. The original or reconstructed image is then passed to warp / depth reprojection processor 1728.

[0159] After warp or depth reprojection, data provided by the eye gaze processor 1730 can be utilized once again to compress the warped image using variable quality encoder 1732 including processor component 1731 that represents image foveation based on eye gaze location. Different foveation processes can be utilized as appropriate to the particular application, including tile-based foveation processes, sparsity-based compression processes, or the like. In some embodiments, processor component 1731 and variable quality encoder 1732 are bypassed. As discussed above, a JPEG encoding process can be performed by variable quality encoder 1732 to form foveated images based on eye gaze, in which the quality of the image varies across the image, providing high quality in the region of the image corresponding to the user's eye gaze and reduced quality in regions of the image more distant from the eye gaze location. Thus, foveated, as well as sparsity encoded images can beformed with reduced size while maintaining desired image quality. The encoded image is then provided to a MIPI device 1734.

[0160] The MIPI device 1734 of wearable 1710 can be connected to MIPI device 1742 of a display system 1740 that includes a processor component 1743 of a variable quality decoder 1744 that represents image defoveation based on eye gaze location and a display device 1746, for example, an LCOS display or a pLED display.

[0161] As illustrated in FIG. 17, the JPEG decoding process can be performed by variable quality decoder 1744 to form final images based on the foveated images produced by variable quality encoder 1732. Thus, embodiments of the present invention reduce system memory and transmission requirements, for example, the amount of data transmitted between the MIPI devices while maintaining a desired image quality. The decoded image is then displayed using display device 1746.

[0162] In some embodiments, variable quality encoder 1732 is bypassed and the warped image is transmitted to the display system 1740 using MIPI device 1734 without variable quality image compression. In these embodiments, the variable quality decoder 1744 is also bypassed.

[0163] FIG. 18 is a simplified block diagram illustrating components of an AR system according to an embodiment of the present invention. AR system 1800 as illustrated in FIG. 18 may be incorporated into the AR devices as described herein. FIG. 18 provides a schematic illustration of one embodiment of AR system 1800 that can perform some or all of the steps of the methods provided by various embodiments. It should be noted that FIG. 18 is meant only to provide a generalized illustration of various components, any or all of which may be utilized as appropriate. FIG. 18, therefore, broadly illustrates how individual system elements may be implemented in a relatively separated or relatively more integrated manner.

[0164] AR system 1800 is shown comprising hardware elements that can be electrically coupled via a bus 1805, or may otherwise be in communication, as appropriate. The hardware elements may include one or more processors 1810, including, without limitation, one or more general-purpose processors and / or one or more special-purpose processors, such as digital signal processing chips, graphics acceleration processors, and / or the like; one or more input devices 1815, which can include, without limitation, a mouse, a keyboard, a camera, and / or the like; and one or more output devices 1820, which can include, without limitation, a display device, a printer, and / or the like. Additionally, AR system 1800 includesan eye tracking system 1855 that can provide the user's eye gaze location to the AR system. Utilizing processor 1810, the foveated image compression techniques discussed herein can be implemented.

[0165] AR system 1800 may further include and / or be in communication with one or more non-transitory storage devices 1825, which can comprise, without limitation, local and / or network accessible storage, and / or can include, without limitation, a disk drive, a drive array, an optical storage device, a solid-state storage device, such as a random access memory (RAM), and / or a read-only memory (ROM), which can be programmable, flash-updateable, and / or the like. Such storage devices may be configured to implement any appropriate data stores, including, without limitation, various file systems, database structures, and / or the like.

[0166] AR system 1800 might also include a communications subsystem 1819, which can include, without limitation, a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or a chipset such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, cellular communication facilities, etc., and / or the like. Communications subsystem 1819 may include one or more input and / or output communication interfaces to permit data to be exchanged with a network such as the network described below, to name one example, other computer systems, television, and / or any other devices described herein. Depending on the desired functionality and / or other implementation concerns, a portable electronic device or similar device may communicate images and / or other information via communications subsystem 1819. In other embodiments, a portable electronic device (e.g., the first electronic device) may be incorporated into AR system 1800 (e.g., an electronic device) as an input device 1815. In some embodiments, AR system 1800 will further comprise a working memory 1860, which can include a RAM or ROM device, as described above

[0167] AR system 1800 also can include software elements, shown as being currently located within working memory 1860, including an operating system 1862, device drivers, executable libraries, and / or other code, such as one or more application programs 1864, which may comprise computer programs provided by various embodiments, and / or may be designed to implement methods, and / or configure systems, provided by other embodiments, as described herein. Merely by way of example, one or more procedures described with respect to the methods discussed above might be implemented as code and / or instructions executable by a computer and / or a processor within a computer; in an aspect, then, such codeand / or instructions can be used to configure and / or adapt a general purpose computer or other device to perform one or more operations in accordance with the described methods.

[0168] A set of these instructions and / or code may be stored on a non-transitory computer- readable storage medium, such as storage device(s) 1825 described above. In some cases, the storage medium might be incorporated within a computer system, such as AR system 1800. In other embodiments, the storage medium might be separate from a computer system, e.g., a removable medium, such as a compact disc, and / or provided in an installation package, such that the storage medium can be used to program, configure, and / or adapt a general purpose computer with the instructions / code stored thereon. These instructions might take the form of executable code, which is executable by AR system 1800 and / or might take the form of source and / or installable code, which, upon compilation and / or installation on AR system 1800, e.g., using any of a variety of generally available compilers, installation programs, compression / decompression utilities, etc., then takes the form of executable code.

[0169] It will be apparent to those skilled in the art that substantial variations may be made in accordance with specific requirements. For example, customized hardware might also be used, and / or particular elements might be implemented in hardware, software including portable software, such as applets, etc., or both. Further, connection to other computing devices, such as network input / output devices, may be employed.

[0170] As mentioned above, in one aspect, some embodiments may employ a computer system such as AR system 1800 to perform methods in accordance with various embodiments of the technology. According to a set of embodiments, some or all of the procedures of such methods are performed by AR system 1800 in response to processor 1810 executing one or more sequences of one or more instructions, which might be incorporated into operating system 1862 and / or other code, such as an application program 1864, contained in working memory 1860. Such instructions may be read into working memory 1860 from another computer-readable medium, such as one or more storage device(s) 1825. Merely by way of example, execution of the sequences of instructions contained in working memory 1860 might cause processor(s) 1810 to perform one or more procedures of the methods described herein. Additionally or alternatively, portions of the methods described herein may be executed through specialized hardware.

[0171] The terms "machine-readable medium" and "computer-readable medium," as used herein, refer to any medium that participates in providing data that causes a machine tooperate in a specific fashion. In an embodiment implemented using AR system 1800, various computer-readable media might be involved in providing instructions / code to processor(s) 1810 for execution and / or might be used to store and / or carry such instructions / code. In many implementations, a computer-readable medium is a physical and / or tangible storage medium. Such a medium may take the form of a non-volatile media or volatile media. Nonvolatile media include, for example, optical and / or magnetic disks, such as storage device(s) 1825. Volatile media include, without limitation, dynamic memory, such as working memory 1860.

[0172] Common forms of physical and / or tangible computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read instructions and / or code.

[0173] Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to processor(s) 1810 for execution. Merely by way of example, the instructions may initially be carried on a magnetic disk and / or optical disc of a remote computer. A remote computer might load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and / or executed by AR system 1800.

[0174] Communications subsystem 1819 and / or components thereof generally will receive signals, and bus 1805 then might carry the signals and / or the data, instructions, etc. carried by the signals to working memory 1860, from which processor(s) 1810 retrieves and executes the instructions. The instructions received by working memory 1860 may optionally be stored on a non-transitory storage device 1825 either before or after execution by processor(s) 1810.

[0175] Various examples of the present disclosure are provided below. As used below, any reference to a series of examples is to be understood as a reference to each of those examples disjunctively (e.g., "Examples 1-4" is to be understood as "Examples 1, 2, 3, or 4").

[0176] Example 1 is a method of displaying a reconstructed image, the method comprising: receiving an image having a plurality of lines of pixel data, wherein each line of the plurality of lines is defined by a plurality of pixel groups; and determining a number of pixels in theimage that are characterized by a pixel brightness less than a predetermined brightness level. If the number of pixels is less than a threshold: forming a first compressed image by compressing the image using a first data compression algorithm; and storing the first compressed image in a memory. If the number of pixels is greater than or equal to a threshold, forming a second compressed image by: forming a mask for each line of the plurality of lines: for each pixel group of the plurality of pixel groups: defining a first bit if pixels in the pixel group are characterized by pixel values less than a brightness threshold; and defining a second bit if pixels in the pixel group are characterized by pixel values greater than or equal to the brightness threshold; providing pixel values for pixels in pixel groups having the second bit; and storing the mask and the provided pixel values for each line in a memory; forming a reconstructed image by: decompressing the first compressed image; or extracting the mask and the provided pixel values for each line from the memory, forming the reconstructed image using the mask and the provided pixel values for each line, and transmitting the reconstructed image to a display.

[0177] Example 2 is the method of example 1 wherein the first data compression algorithm comprises DSC.

[0178] Example 3 is the method of example(s) 1-2 wherein the first data compression algorithm comprises VDC-X.

[0179] Example 4 is the method of example(s) 1 -3 wherein the first bit is " 1 " and the second bit is "0".

[0180] Example 5 is the method of example(s) 1-4 wherein the mask includes N bits, each line includes M pixel groups, and each line includes N x M pixels.

[0181] Example 6 is the method of example(s) 1-5 wherein the provided pixel values do not include pixel values for pixels in pixel groups having the first bit.

[0182] Example 7 is the method of example(s) 1-6 wherein forming the reconstructed image comprises: for each line: for each pixel group: defining pixel values as black pixels if the first bit is defined for the pixel group; and defining pixel values as the provided pixel values for pixels in the pixel group if the second bit is defined for the pixel group.

[0183] Example 8 is the method of example(s) 1-7 wherein: forming the first compressed image by compressing the image using a first data compression algorithm comprises: determining an eye gaze location of a user; compressing a primary quality region of theimage using a compression ratio of X — > Y bits per pixel (bpp) to form a primary quality image; compressing the image using a compression ratio of X — > Z bpp, wherein Z < Y to form a secondary quality image; and decompressing the first compressed image comprises overlaying the primary quality image on the secondary quality image.

[0184] Example 9 is the method of example(s) 1-8 wherein X = 24, Y = 15 and Z = 12.

[0185] Example 10 is the method of example(s) 1-9 wherein X = 24, Y = 15 and Z = 10.

[0186] Example 11 is the method of example(s) 1-10 wherein X = 24, Y = 15 and Z = 8.

[0187] Example 12 is the method of example(s) 1-11 wherein X = 24, Y = 15 and Z = 6.

[0188] Example 13 is the method of example(s) 1-12 wherein the primary quality region includes the eye gaze location.

[0189] Example 14 is the method of example(s) 1-13 wherein the primary quality region has a rectangular shape centered at the eye gaze location.

[0190] Example 15 is the method of example(s) 1-14 wherein the image comprises an M x N image and the primary quality region comprises an aM x aN image, where a<l.

[0191] Example 16 is the method of example(s) 1-15 wherein forming the reconstructed image comprises defining pixels in pixel groups having the first bit as black pixels.

[0192] Example 17 is the method of example(s) 1-16 wherein the image comprises virtual content.

[0193] Example 18 is the method of example(s) 1-17 wherein the virtual content is received at a first frame rate and the reconstructed image is displayed at a second frame rate higher than the first frame rate.

[0194] Example 19 is the method of example(s) 1-18 further comprising performing image processing on the reconstructed image prior to displaying the reconstructed image.

[0195] Example 20 is the method of example(s) 1-19 further comprising warping the reconstructed image prior to transmitting the reconstructed image to the display.

[0196] Example 21 is the method of example(s) 1-20 wherein: forming the first compressed image by compressing the image using a first data compression algorithm comprises: dividing the image into N sections; determining an eye gaze location of a user; compressing N-M sections using a compression ratio of X — > Y bpp to form a primary quality image;compressing M sections using a compression ratio of X — > Z bpp, wherein Z < Y to form a secondary quality image; and decompressing the first compressed image comprises overlaying the primary quality image on the secondary quality image.

[0197] Example 22 is the method of example(s) 1-21 wherein N = 16 and M = 12.

[0198] Example 23 is the method of example(s) 1-22 wherein X = 24, Y = 15 and Z = 12.

[0199] Example 24 is the method of example(s) 1-23 wherein X = 24, Y = 15 and Z = 10.

[0200] Example 25 is the method of example(s) 1-24 wherein X = 24, Y = 15 and Z = 8.

[0201] Example 26 is the method of example(s) 1-25 wherein X = 24, Y = 15 and Z = 6.

[0202] Example 27 is the method of example(s) 1-26 wherein the primary quality image includes the eye gaze location.

[0203] Example 28 is the method of example(s) 1- 27 wherein the primary quality image has a rectangular shape centered at the eye gaze location.

[0204] Example 29 is the method of example(s) 1-28 wherein the image comprises an M x N image and the primary quality image comprises an aM x aN image, where a<l.

[0205] Example 30 is an augmented reality (AR) system comprising: a wearable device including: a frame; a display coupled to the frame; and an eye tracking system; a memory; and a processor configured to execute the method of example 1.

[0206] Example 31 is a method of displaying a reconstructed image, the method comprising: receiving an image; determining an eye gaze location of a user; compressing a primary quality region of the image using a compression ratio of X — > Y bits per pixel (bpp) to form a primary quality image; compressing a secondary quality region of the image using a compression ratio of X — > Z bpp, wherein Z < Y to form a secondary quality image; overlaying the primary quality image on the secondary quality image to form the reconstructed image; and displaying the reconstructed image on a display.

[0207] Example 32 is the method of example 31 wherein the secondary quality region of the image consists of the image.

[0208] Example 33 is the method of example(s) 31-32 wherein determining the eye gaze location comprises use of an eye tracking camera of an augmented reality device.

[0209] Example 34 is the method of example(s) 31-33 wherein the image comprises virtual content generated by an augmented reality device.

[0210] Example 35 is the method of example(s) 31-34 wherein the virtual content is received at a first frame rate and the reconstructed image is displayed at a second frame rate higher than the first frame rate.

[0211] Example 36 is the method of example(s) 31-35 wherein X = 24, Y = 15 and Z = 12.

[0212] Example 37 is the method of example(s) 31-36 wherein X = 24, Y = 15 and Z = 10.

[0213] Example 38 is the method of example(s) 31-37 wherein X = 24, Y = 15 and Z = 8.

[0214] Example 39 is the method of example(s) 31-38 wherein X = 24, Y = 15 and Z = 6.

[0215] Example 40 is the method of example(s) 31-39 further comprising post-processing image content in at least one of the primary quality region or the secondary quality region.

[0216] Example 41 is the method of example(s) 31-40 wherein the primary quality image includes the eye gaze location.

[0217] Example 42 is the method of example(s) 31-41 wherein the primary quality image has a rectangular shape centered at the eye gaze location.

[0218] Example 43 is the method of example(s) 31-42 wherein the image comprises an M x N image and the primary quality image comprises an aM x aN image, where a<l.

[0219] Example 44 is the method of example(s) 31-43 further comprising performing image processing on the reconstructed image prior to displaying the reconstructed image.

[0220] Example 45 is the method of example(s) 31-44 further comprising: warping the reconstructed image; and transmitting the reconstructed image to the display.

[0221] Example 46 is an augmented reality (AR) system comprising: a wearable device including: a frame; a display coupled to the frame; and an eye tracking system; a memory; and a processor configured to: receive an image; determine, using the eye tracking system, an eye gaze location of a user; compress a primary quality region of the image using a compression ratio of X — > Y bits per pixel (bpp) to form a primary quality image; compress a secondary quality region of the image using a compression ratio of X — > Z bpp, wherein Z < Y to form a secondary quality image; overlay the primary quality image on the secondaryquality image to form a reconstructed image; and display the reconstructed image on the display.

[0222] Example 47 is a method of displaying a reconstructed image, the method comprising: receiving an image; dividing the image into N sections; determining an eye gaze location of a user; compressing N-M sections using a compression ratio of X — > Y bits per pixel (bpp) to form a primary quality image; compressing M sections using a compression ratio of X — > Z bpp, wherein Z < Y to form a secondary quality image; and overlaying the primary quality image on the secondary quality image to form the reconstructed image; and displaying the reconstructed image on a display.

[0223] Example 48 is the method of example 47 wherein compressing the N-M sections using a compression ratio X — > Y bpp comprises compressing all blocks in the N-M sections using the compression ratio of X — > Y bpp.

[0224] Example 49 is the method of example(s) 47-48 wherein compressing the M sections using a compression ratio X — > Z bpp comprises compressing all blocks in the M sections using the compression ratio of X — > Z bpp.

[0225] Example 50 is the method of example(s) 47-49 wherein determining the eye gaze location comprises use of an eye tracking camera of an augmented reality device.

[0226] Example 51 is the method of example(s) 47-50 wherein the image comprises virtual content generated by an augmented reality device.

[0227] Example 52 is the method of example(s) 47-51 wherein the virtual content is received at a first frame rate and the reconstructed image is displayed at a second frame rate higher than the first frame rate.

[0228] Example 53 is the method of example(s) 47-52 wherein X = 24, Y = 15 and Z = 12.

[0229] Example 54 is the method of example(s) 47-53 wherein X = 24, Y = 15 and Z = 10.

[0230] Example 55 is the method of example(s) 47-54 wherein X = 24, Y = 15 and Z = 8.

[0231] Example 56 is the method of example(s) 47-55 wherein X = 24, Y = 15 and Z = 6.

[0232] Example 57 is the method of example(s) 47-56 further comprising post-processing image content in at least one of the primary quality image or the secondary quality image.

[0233] Example 58 is the method of example(s) 47-57 wherein at least one of the N-M sections includes the eye gaze location.

[0234] Example 59 is the method of example(s) 47-58 wherein the image comprises an M x N image and the primary quality image comprises an aM x aN image, where a<l.

[0235] Example 60 is the method of example(s) 47-59 further comprising performing image processing on the reconstructed image prior to displaying the reconstructed image.

[0236] Example 61 is the method of example(s) 47-60 further comprising: warping the reconstructed image; and transmitting the reconstructed image to the display.

[0237] Example 62 is an augmented reality (AR) system comprising: a wearable device including: a frame; a display coupled to the frame; and an eye tracking system; a memory; and a processor configured to: receive an image; divide the image into N sections; determine, using the eye tracking system, an eye gaze location of a user; compress N-M sections using a compression ratio of X — > Y bits per pixel (bpp) to form a primary quality image; compress M sections using a compression ratio of X — > Z bpp, wherein Z < Y to form a secondary quality image; and overlay the primary quality image on the secondary quality image to form a reconstructed image; and display the reconstructed image on the display.

[0238] Accordingly, the claims are not intended to be limited to the embodiments shown herein, but are to be accorded the widest scope consistent with this disclosure, the principles and the novel features disclosed herein. Thus, it is also understood that the examples and embodiments described herein are for illustrative purposes only and that various modifications or changes in light thereof will be suggested to persons skilled in the art and are to be included within the spirit and purview of this application and scope of the appended claims.

Claims

WHAT IS CLAIMED IS:

1. A method of displaying a reconstructed image, the method comprising: receiving an image having a plurality of lines of pixel data, wherein each line of the plurality of lines is defined by a plurality of pixel groups; determining a number of pixels in the image that are characterized by a pixel brightness less than a predetermined brightness level; if the number of pixels is less than a threshold: forming a first compressed image by compressing the image using a first data compression algorithm; and storing the first compressed image in a memory; if the number of pixels is greater than or equal to a threshold, forming a second compressed image by: forming a mask for each line of the plurality of lines: for each pixel group of the plurality of pixel groups: defining a first bit if pixels in the pixel group are characterized by pixel values less than a brightness threshold; and defining a second bit if pixels in the pixel group are characterized by pixel values greater than or equal to the brightness threshold; providing pixel values for pixels in pixel groups having the second bit; and storing the mask and the provided pixel values for each line in a memory; forming a reconstructed image by: decompressing the first compressed image; or extracting the mask and the provided pixel values for each line from the memory, forming the reconstructed image using the mask and the provided pixel values for each line, and transmitting the reconstructed image to a display.

2. The method of claim 1 wherein the first data compression algorithm comprises DSC.

3. The method of claim 1 wherein the first data compression algorithm comprises VDC-X.

4. The method of claim 1 wherein the first bit is " 1 " and the second bit is5. The method of claim 1 wherein the mask includes N bits, each line includes M pixel groups, and each line includes N x M pixels.

6. The method of claim 1 wherein the provided pixel values do not include pixel values for pixels in pixel groups having the first bit.

7. The method of claim 1 wherein forming the reconstructed image comprises: for each line: for each pixel group: defining pixel values as black pixels if the first bit is defined for the pixel group; and defining pixel values as the provided pixel values for pixels in the pixel group if the second bit is defined for the pixel group.

8. The method of claim 1 wherein: forming the first compressed image by compressing the image using a first data compression algorithm comprises: determining an eye gaze location of a user; compressing a primary quality region of the image using a compression ratio of X — > Y bits per pixel (bpp) to form a primary quality image; compressing the image using a compression ratio of X — > Z bpp, wherein Z < Y to form a secondary quality image; and decompressing the first compressed image comprises overlaying the primary quality image on the secondary quality image.

9. The method of claim 8 wherein X = 24, Y = 15 and Z = 12.

10. The method of claim 8 wherein X = 24, Y = 15 and Z = 10.

11. The method of claim 8 wherein X = 24, Y = 15 and Z = 8.

12. The method of claim 8 wherein X = 24, Y = 15 and Z = 6.

13. The method of claim 8 wherein the primary quality region includes the eye gaze location.

14. The method of claim 13 wherein the primary quality region has a rectangular shape centered at the eye gaze location.

15. The method of claim 8 wherein the image comprises an M x N image and the primary quality region comprises an aM x aN image, where a<l.

16. The method of claim 1 wherein forming the reconstructed image comprises defining pixels in pixel groups having the first bit as black pixels.

17. The method of claim 1 wherein the image comprises virtual content.

18. The method of claim 17 wherein the virtual content is received at a first frame rate and the reconstructed image is displayed at a second frame rate higher than the first frame rate.

19. The method of claim 1 further comprising performing image processing on the reconstructed image prior to displaying the reconstructed image.

20. The method of claim 1 further comprising warping the reconstructed image prior to transmitting the reconstructed image to the display.

21. The method of claim 1 wherein: forming the first compressed image by compressing the image using a first data compression algorithm comprises: dividing the image into N sections; determining an eye gaze location of a user; compressing N-M sections using a compression ratio of X — > Y bpp to form a primary quality image; compressing M sections using a compression ratio of X — > Z bpp, wherein Z < Y to form a secondary quality image; and decompressing the first compressed image comprises overlaying the primary quality image on the secondary quality image.

22. The method of claim 21 wherein N = 16 and M = 12.

23. The method of claim 21 wherein X = 24, Y = 15 and Z = 12.

24. The method of claim 21 wherein X = 24, Y = 15 and Z = 10.

25. The method of claim 21 wherein X = 24, Y = 15 and Z = 8.

26. The method of claim 21 wherein X = 24, Y = 15 and Z = 6.

27. The method of claim 21 wherein the primary quality image includes the eye gaze location.

28. The method of claim 27 wherein the primary quality image has a rectangular shape centered at the eye gaze location.

29. The method of claim 21 wherein the image comprises an M x N image and the primary quality image comprises an aM x aN image, where a<l.

30. An augmented reality (AR) system comprising: a wearable device including: a frame; a display coupled to the frame; and an eye tracking system; a memory; and a processor configured to execute the method of claim 1.

31. A method of displaying a reconstructed image, the method comprising: receiving an image; determining an eye gaze location of a user; compressing a primary quality region of the image using a compression ratio of X — Y bits per pixel (bpp) to form a primary quality image; compressing a secondary quality region of the image using a compression ratio of X — > Z bpp, wherein Z < Y to form a secondary quality image; overlaying the primary quality image on the secondary quality image to form the reconstructed image; and displaying the reconstructed image on a display.

32. The method of claim 31 wherein the secondary quality region of the image consists of the image.

33. The method of claim 31 wherein determining the eye gaze location comprises use of an eye tracking camera of an augmented reality device.

34. The method of claim 31 wherein the image comprises virtual content generated by an augmented reality device.

35. The method of claim 34 wherein the virtual content is received at a first frame rate and the reconstructed image is displayed at a second frame rate higher than the first frame rate.

36. The method of claim 31 wherein X = 24, Y = 15 and Z = 12.

37. The method of claim 31 wherein X = 24, Y = 15 and Z = 10.

38. The method of claim 31 wherein X = 24, Y = 15 and Z = 8.

39. The method of claim 31 wherein X = 24, Y = 15 and Z = 6.

40. The method of claim 31 further comprising post-processing image content in at least one of the primary quality region or the secondary quality region.

41. The method of claim 31 wherein the primary quality image includes the eye gaze location.

42. The method of claim 31 wherein the primary quality image has a rectangular shape centered at the eye gaze location.

43. The method of claim 31 wherein the image comprises an M x N image and the primary quality image comprises an aM x aN image, where a<l.

44. The method of claim 31 further comprising performing image processing on the reconstructed image prior to displaying the reconstructed image.

45. The method of claim 31 further comprising: warping the reconstructed image; andtransmitting the reconstructed image to the display.

46. An augmented reality (AR) system comprising: a wearable device including: a frame; a display coupled to the frame; and an eye tracking system; a memory; and a processor configured to: receive an image; determine, using the eye tracking system, an eye gaze location of a user; compress a primary quality region of the image using a compression ratio of X — Y bits per pixel (bpp) to form a primary quality image; compress a secondary quality region of the image using a compression ratio of X — > Z bpp, wherein Z < Y to form a secondary quality image; overlay the primary quality image on the secondary quality image to form a reconstructed image; and display the reconstructed image on the display.

47. A method of displaying a reconstructed image, the method comprising: receiving an image; dividing the image into N sections; determining an eye gaze location of a user; compressing N-M sections using a compression ratio of X — > Y bits per pixel (bpp) to form a primary quality image; compressing M sections using a compression ratio of X — > Z bpp, wherein Z < Y to form a secondary quality image; and overlaying the primary quality image on the secondary quality image to form the reconstructed image; and displaying the reconstructed image on a display.

48. The method of claim 47 wherein compressing the N-M sections using a compression ratio X — > Y bpp comprises compressing all blocks in the N-M sections using the compression ratio of X — > Y bpp.

49. The method of claim 47 wherein compressing the M sections using a compression ratio X — > Z bpp comprises compressing all blocks in the M sections using the compression ratio of X — > Z bpp.

50. The method of claim 47 wherein determining the eye gaze location comprises use of an eye tracking camera of an augmented reality device.

51. The method of claim 47 wherein the image comprises virtual content generated by an augmented reality device.

52. The method of claim 51 wherein the virtual content is received at a first frame rate and the reconstructed image is displayed at a second frame rate higher than the first frame rate.

53. The method of claim 47 wherein X = 24, Y = 15 and Z = 12.

54. The method of claim 47 wherein X = 24, Y = 15 and Z = 10.

55. The method of claim 47 wherein X = 24, Y = 15 and Z = 8.

56. The method of claim 47 wherein X = 24, Y = 15 and Z = 6.

57. The method of claim 47 further comprising post-processing image content in at least one of the primary quality image or the secondary quality image.

58. The method of claim 47 wherein at least one of the N-M sections includes the eye gaze location.

59. The method of claim 47 wherein the image comprises an M x N image and the primary quality image comprises an aM x aN image, where a<l.

60. The method of claim 47 further comprising performing image processing on the reconstructed image prior to displaying the reconstructed image.

61. The method of claim 47 further comprising: warping the reconstructed image; and transmitting the reconstructed image to the display.

62. An augmented reality (AR) system comprising: a wearable device including: a frame; a display coupled to the frame; and an eye tracking system; a memory; and a processor configured to: receive an image; divide the image into N sections; determine, using the eye tracking system, an eye gaze location of a user; compress N-M sections using a compression ratio of X — > Y bits per pixel (bpp) to form a primary quality image; compress M sections using a compression ratio of X — > Z bpp, wherein Z < Y to form a secondary quality image; and overlay the primary quality image on the secondary quality image to form a reconstructed image; and display the reconstructed image on the display.

Images