Near-eye display and computing device

Abstract

A Near-Eye Display (NED) (100) comprising a display element with equidistant pixels and a computing device (200) are disclosed. The NED (100) is operative to receive (425) encoded image frames from the computing device (200), the received image frames having different image quality for at least two regions of the image frames corresponding to regions of a Field-of-View of the NED (100) having different average effective display quality, decode (417) the encoded image frames, and display (418) the decoded image frames using the display element. The computing device (200) is operative to encode (424) the image frames with different image quality for at least two regions of the image frames, and send (425) the encoded image frames to the NED.

Description

[0001] NEAR-EYE DISPLAY AND COMPUTING DEVICE

[0002] Technical field

[0003] The invention relates to a Near-Eye Display (NED) comprising a display element with equidistant pixels, a computing device for rendering image frames for display by a NED comprising a display element with equidistant pixels, a method performed by a NED comprising a display element with equidistant pixels, a method of rendering image frames for display by a NED comprising a display element with equidistant pixels, corresponding computer programs, corresponding computer-readable data carriers, and corresponding data carrier signals.

[0004] Background

[0005] Virtual Reality (VR) involves rendering a virtual environment that a user can view and interact with. Augmented Reality (AR) involves combining the real world with virtual environments, to enhance or change a user’s view of the real world. For example, virtual objects can be displayed as an overlay onto the user’s view of the real world. Mixed Reality (MR) is an extension of AR which allows real-world objects and virtual objects to interact. Extended Reality (XR) is often used as an umbrella term for VR, AR, and MR.

[0006] The user interacts with an XR environment via a Head-Mounted Display (HMD) or smart glasses which allow the user to view a 3D representation of the virtual environment, optionally (e.g., for AR) as an overlay on top of the user’s view of the real world. Current commercial HMDs comprise one or two Near-Eye Displays (NEDs) which create virtual images in the Field-of-View (FoV) of one or both eyes of the user wearing the HMD. To the eye, the virtual image appears at a distance and much larger than the relatively small display panel used to display the image. Many NEDs rely on a positive lens which is placed closer to the display element than the focal length of the lens. By adjusting the placement of the lens relative to the display element, the distance at which the user perceives a virtual object to be located in the real- world can be selected. Today’s NEDs allow viewing images comfortably even though all physical parts, such as the display element and one or more optical elements, are placed within a few cm from the user’s eyes.

[0007] Throughout this disclosure, the terms HMD and NED are used interchangeably. In other words, a NED is understood to not only comprise a display element and one or more optical elements, but to also comprise processing circuitry and other components, e.g., a communications interface circuitry for exchanging data with a separate computing device over a wired or wireless connection.

[0008] Because of the limited computing and battery resources of NEDs, the virtual content which is displayed by the display elements is typically rendered by computing devices which are separate from the NEDs, e.g., a companion device worn by the user, an edge cloud, or an application server. The image frames rendered by the computing device are transmitted to the NED for display, often over a wireless connection. In order to reduce the bandwidth required for transmitting the rendered image frames to the NED, the image frames are encoded by the computing device before transmission to the NED, and subsequently decoded by the NED for display, using a known video coding technology.

[0009] A known issue of video coding is the trade-off between image quality and required bandwidth, i.e. , a higher image quality requires a higher bandwidth. Known video coding technologies are designed for flat-panel displays, which have a substantially constant display quality across their FoV.

[0010] It is an object of the invention to provide an improved alternative to the above techniques and prior art.

[0011] More specifically, it is an object of the invention to provide improved solutions for transmitting image frames which are rendered and encoded by a computing device for transmission to, and subsequent display by, a NED. In particular, it is an object of the invention to provide solutions which transmit encoded images frames more efficiently.

[0012] These and other objects of the invention are achieved by means of different aspects of the invention, as defined by the independent claims. Embodiments of the invention are characterized by the dependent claims.

[0013] According to a first aspect of the invention, a NED comprising a display element with equidistant pixels is provided. The NED is operative to receive encoded image frames from a computing device. The received image frames have different image quality for at least two regions of the image frames. The at least two regions of the image frames correspond to regions of an FoV of the NED having different average effective display quality. The NED is further operative to decode the encoded image frames. The NED is further operative to display the decoded image frames using the display element.

[0014] According to a second aspect of the invention, a computing device for rendering image frames for display by a NED is provided. The NED comprises a display element with equidistant pixels. The computing device is operative to encode the image frames with different image quality for at least two regions of the image frames. The computing device is further operative to send the encoded image frames to the NED.

[0015] According to a third aspect of the invention, a method is provided. The method is performed by a NED comprising a display element with equidistant pixels. The method comprises receiving encoded image frames from a computing device. The received image frames have different image quality for at least two regions of the image frames. The at least two regions of the image frames correspond to regions of an FoV of the NED having different average effective display quality. The method further comprises decoding the encoded image frames. The method further comprises displaying the decoded image frames using the display element.

[0016] According to a fourth aspect of the invention, a method of rendering image frames for display by a NED is provided. The NED comprises a display element with equidistant pixels. The method is performed by a computing device and comprises encoding the image frames with different image quality for at least two regions of the image frames. The method further comprises sending the encoded image frames to the NED.

[0017] According to a fifth aspect of the invention, a computer program is provided. The computer program comprises instructions which, when the computer program is executed by one or more processors comprised in a NED, cause the NED to carry out the method according to an embodiment of the third aspect of the invention.

[0018] According to a sixth aspect of the invention, a computer program is provided. The computer program comprises instructions which, when the computer program is executed by one or more processors comprised in a computing device, cause the computing device to carry out the method according to an embodiment of the fourth aspect of the invention.

[0019] The invention makes use of an understanding that image frames which are rendered by a computing device and transmitted to a NED for display to a user do not need to be encoded with the same image quality over the entire region of the image frames. Rather, since the effective display quality varies over the FoV of the NED, the images frames may be divided into at least two regions having different image quality. That is, parts of the image frames, typically the central parts, are encoded with higher image quality, to match the higher effective display quality of the NED close to the optical axis, whereas peripheral parts of the image frames are encoded with lower image quality.

[0020] Even though advantages of the invention have in some cases been described with reference to embodiments of the first and second aspect of the invention, corresponding reasoning applies to embodiments of other aspects of the invention.

[0021] Further objectives of, features of, and advantages with, the invention will become apparent when studying the following detailed disclosure, the drawings, and the appended claims. Those skilled in the art realize that different features of the invention can be combined to create embodiments other than those described in the following.

[0022] Brief description of the drawings

[0023] The above, as well as additional objects, features and advantages of the invention, will be better understood through the following illustrative and non-limiting detailed description of embodiments of the invention, with reference to the appended drawings, in which:

[0024] Fig. 1 schematically illustrates a NED, in accordance with embodiments of the invention.

[0025] Fig. 2 schematically illustrates a computing device, in accordance with embodiments of the invention.

[0026] Fig. 3 schematically illustrates an image frame divided into two regions, in accordance with embodiments of the invention.

[0027] Fig. 4 shows a sequence diagram illustrating embodiments of the invention.

[0028] Fig. 5 shows a flow chart illustrating a method performed by a NED comprising a display element with equidistant pixels, in accordance with embodiments of the invention. Fig. 6 shows a flow chart illustrating a method of rendering image frames for display by a NED comprising a display element with equidistant pixels, in accordance with embodiments of the invention.

[0029] All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary in order to elucidate the invention, wherein other parts may be omitted or merely suggested.

[0030] Detailed description

[0031] The invention will now be described more fully herein after with reference to the accompanying drawings, in which certain embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0032] HMDs or other types of XR display devices, such as smart glasses, which users can utilize to view a VR or an AR environment typically comprise one or two NEDs. With reference to Fig. 1 , a NED 100 comprises a display element 101 for displaying image frames, either still images, a video, or a stream of images frames or a video, for viewing by a user (using his / her eye 110) through one or more optical elements 102, such as lenses, mirrors, waveguides, etc. Different designs of HMDs are known in the art. For instance, an HMD may comprise a single display element and a single set of optical elements for both eyes of the user. Alternatively, an HMD may comprise a single display element in combination with two sets of optical elements, one for each eye. The most common configuration in today’s HMDs comprises two separate NEDs, each NED comprising a display element and one or more optical elements, one for each eye. However, even though most HMDs are designed for binocular vision, embodiments of the invention may be envisaged to provide monocular vision, e.g., an HMD or smart glass having a single NED 100, i.e. , a single set of a display element 101 and one or more optical elements 102, for one eye only.

[0033] In the following, the terms HMD and NED are used interchangeably.

[0034] As discussed above, HMDs may be used for VR or AR applications. In VR, the user only views virtual content which is rendered by a computing device which is communicatively connected with the NED. In AR, on the other hand, virtual content which is generated by a computing device is overlaid onto a view of the real-world scene in front of the user. The latter may be accomplished by virtue of a front-facing camera capturing the real- world scene in front of the user, and overlaying or combining virtual content with captured real-world scene for display by the display element. As an alternative HMDs based on Optical See-Through (OST) displays allow the user to directly view the real-world scene in front of the user, with virtual content being displayed by the display element such that it appears to be overlaid onto the view of the real-world scene. This is achieved by utilizing optical combiners, e.g., half-mirrors, waveguides, or the like.

[0035] A common issue with conventional NEDs is that the effective display quality, which is perceived by a user viewing the display element with his / her eye(s), varies over the FoV of the NED. Display quality can be described or quantified as sharpness, contrast, resolution, or a combination thereof. For example, and with reference to Fig. 1 , NEDs such as the NED 100 relying on a positive lens 102 for magnifying an image frame displayed by the display element 101 suffer from negative effects on the display quality in the peripheral parts of the displayed image frame, away from the optical center (in Fig. 1 indicated by dashed-dotted line 120). In the present context, the effective display quality is the display quality, such as sharpness, contrast, and / or resolution, perceived by the user viewing the display element 101 (i.e., the image frames displayed by the display element 101 ), which can be achieved by the combination of the display element 101 and the optics, such as the lens 102. In other words, the effective display quality is the upper bound for the image quality which can be achieved by displaying an image frame using the display element 101 . It is typically limited by the resolution and other properties of the display element 101 , as well as the optical elements such as lenses 102, waveguides, or mirrors.

[0036] Because of the limited computing and battery resources of NEDs (in Fig. 1 illustrated as processing circuitry 103 comprised in the NED 100), the virtual content which is displayed by the display elements 101 of the NEDs 100 may be rendered by separate computing devices, such as the computing device 200 which is illustrated in Fig. 2. The computing device 200 may, e.g., be a companion device worn by the user, an edge cloud, or an application server. The image frames rendered by the computing device 200 are transmitted to the NED 100 for display, typically over a wireless connection, by virtue of communication interface circuitries 107 and 207 in the HMD 100 and the computing device 200, respectively, using one or more communications technology such as Bluetooth, a cellular technology, Wi- Fi / WLAN, Universal Serial Bus (USB), etc.

[0037] In order to reduce the bandwidth required for transmitting the rendered image frames from the computing device 200 to the NED 100, the image frames are encoded (or compressed) by the computing device 200 before transmission to the NED 100, and subsequently decoded (using a reverse process) by the NED 100 for display using the display element 101. This may be achieved using known video coding technologies such as High Efficiency Video Coding (HEVC, aka H.265), Versatile Video Coding (VVC, aka H.266), or the like.

[0038] A known issue of video coding is the trade-off between image quality and required bandwidth, since a higher image quality requires a higher bandwidth. For NEDs which have a spatially varying effective display quality over their FoV, encoding image frames with a high image quality to match the maximum effective display quality in the central region of the FoV of the NED, would result in a high effective image quality also in the peripheral regions of the FoV where the effective display quality is lower. This would lead to a relatively high bitrate, and accordingly required bandwidth, to support a high image quality over the entire FoV of the NED. However, since the effective display quality of the NED in the peripheral region in practice limits the quality of the displayed image frame which the user can perceive, the bandwidth of the (wireless) connection between the computing device 200 and the NED 100 is not utilized efficiently. The user viewing the displayed image with his / her eye(s) 110 frame can simply not perceive the high image quality of the displayed image frame, except in the central region of the image frames, corresponding to the central region of the FoV.

[0039] In the following, embodiments of a NED 100 are described with reference to Fig. 1 . Further reference is made to Fig. 3, which schematically illustrates an image frame divided into two regions, in accordance with embodiments of the invention, and Fig. 4, which shows a sequence diagram illustrating embodiments of the invention.

[0040] The NED 100 may, e.g., be, or may be comprised in, an HMD, smart glasses, an VR / AR / MR / XR headset, or the like. The NED 100 comprises a display element 101 with equidistant pixels. The display element 101 may be based on Light-Emitting Diodes (LEDs), Organic LEDs (OLEDs), quantumdot based LEDs, or Liquid Crystal Displays (LCDs). A display element with equidistant pixels typically has pixels of equal size. In combination with one or more optical elements (in Fig. 1 illustrated as the lens 102), the equidistant pixels of the display element 101 are perceived by a user viewing the display element with his / her eye(s) 110 as pixels having a substantially constant distribution of pixels (aka pixel-per-degree) across the FoV of the NED 100.

[0041] The NED 100 may comprise a positive lens 102 at a distance to the display element 101 which is smaller than a focal length of the lens 102. However, embodiments of the invention are not limited to NEDs comprising a single positive lens 102 for each display element 101 . Rather, NEDs may be envisaged which rely on one or more optical elements such as lenses, waveguides, mirrors, or a combination thereof, for magnifying image frames displayed by the display element 101 for projection towards a user’s (or viewer’s) eye(s) 110. Due to imperfections in the optical elements 102, and / or fundamental limitations in the optical design of NEDs, the effective display quality, sharpness, contrast, resolution, or a combination thereof, typically decreases with distance from an optical axis 120 of the NED 100. However, one may envisage NED designs which exhibit a spatially varying effective display quality which is not maximum at the center of the FoV and decreases with increasing (radial) distance from the optical axis 120.

[0042] As is illustrated in Fig. 1 , the NED 100 further comprises a processing circuitry 103 which causes the NED 100 to become operative in accordance with embodiments of the invention disclosed herein. The processing circuitry 103 may comprise one or more processors 104, such as Central Processing Units (CPUs), microprocessors, application processors, application-specific processors, Graphics Processing Units (GPUs), Digital Signal Processors (DSPs) including image processors, video codecs, or a combination thereof, and a memory 105 comprising a computer program 106, i.e., software, comprising instructions. When executed by the processor(s) 104, the instructions cause the NED 100 to become operative in accordance with embodiments of the invention disclosed herein. The processing circuitry 103 may alternatively or additionally comprise one or more Application-Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or the like, which are operative to cause the NED 100 to become operative in accordance with embodiments of the invention disclosed herein.

[0043] More specifically, the NED 100 is operative to receive 425 encoded image frames from a computing device 200. The computing device 200, which is described in further detail below, and with reference to Fig. 2, may, e.g., be a companion device worn by the user, an edge server, an application server, or the like. The computing device 200 is typically more powerful, or less constrained, in terms of computing and battery resources as the NED 100. The received images frames may be still images, part of a video, or part of a stream of image frames or a video.

[0044] The NED 100 may be operative to receive 425 the encoded image frames from the computing device 200 via a wireless connection. Such a wireless connection may commence between a communications interface circuitry 107 comprised in the NED 100, and a corresponding communications interface circuitry 207 comprised in the computing device 200. The communications interface circuitry 107 may, e.g., be a cellular modem, a Bluetooth modem, a Wi-Fi / WLAN modem, an Ethernet modem, or the like, and operative to exchange date, in particular a stream of encoded image frames and optionally supplementary information, as described herein, between the NED 100 and the computing device 200.

[0045] The received 425 image frames have different image quality for at least two regions of the image frames. These two regions of the image frames correspond to regions of the display element 101 having different average effective display quality across the FoV of the NED 100. In practice, this mean that the images frames have been encoded 424, by the computing device 200, before transmission to the NED 100, with different settings or parameters for the at least two regions of the image frames, as is described in further detail below.

[0046] Optionally, the at least two regions of the image frames may comprise a central region having the highest image quality. This is exemplified in Fig. 3 which schematically illustrates an image frame 300, corresponding to an active (i.e., light-emitting) area of the display element 101 , divided into two regions. Here, the central region 301 is illustrated to be circular, surrounded by a peripheral region 302 (covering the remainder of the image frame 300). In this example, the circular central region 301 may be encoded with a higher image quality owing to the higher effective display quality in the central part of the FoV of the NED 100, close to the optical axis 120 of the NED 100. However, different from what is illustrated in Fig. 3, the regions need not to be circular but may, e.g., be square or rectangular. It will also be appreciated that embodiments of the invention may rely on more than two regions of the image frames, the regions being encoded with different image qualities.

[0047] The NED 100 is further operative to decode 417 the encoded image frames, using the same or a corresponding (the reverse process) video coding technology used for encoding 424 the image frames before transmission 425 to the NED 100.

[0048] The NED 100 is further operative to display 418 the decoded image frames using the display element 101 for viewing by the user.

[0049] The NED 100 may further be operative to send 411 information pertaining to a spatial variation of the effective display quality over an FoV of the NED 100 to the computing device 200. For example, the information pertaining to a spatial variation of the effective display quality may be represented by a map, or represented by a mathematical function, e.g., as a function of radial distance from the optical axis 120. The information may either pertain to a quantified display quality, e.g., sharpness, or to a relative display quality, e.g., indicating a change in display quality or coding efficiency relative to a central region 301 having the highest quality. The information may be generic to a type or model of the NED 100, or specific, e.g., obtained by measuring the effective display quality of the NED 100 as part of a calibration procedure. As an alternative to sending 411 a map or other representation of the spatial variation of the effective display quality to the computing device 200, the NED 100 may send 411 information which enables the computing device 200 to retrieve 441 such information from a repository 400. For example, the information pertaining to a spatial variation of the effective display quality may, e.g., be a Uniform Resource Locator (URL) or other hyperlink, pointing to a repository 400, a model identifier, an identifier of the NED 100, or the like. Advantageously, sending 411 the information pertaining to a spatial variation of the effective display quality over an FoV of the NED 100 to the computing device 200 enables the computing device 200 to select the at least two regions of the image frames and / or the image quality for the at least two regions of the image frames.

[0050] In the present context, selecting the at least two regions of the image frames, such as the regions 301 and 302 sketched in Fig. 3, is to be understood as partitioning the entire area 300 of an image frame, corresponding to an active area of the display element 101 where the image frames are displayed, into two or more regions which preferably are nonoverlapping and together cover the entire area of the image frame. The partitioning of the entire area of an image frame into two or more regions may, e.g., be signaled by the NED 100 to the computing device 200 by indicating one or more blocks, tiles, or slices, which together make up a region. As I described further below, blocks, tiles, or slices, are concepts known from video codes such as HEVC and VVC to partition image frames into structural units which can be decoded with different image quality.

[0051] Further, selecting the image quality for the at least two regions is to be understood as selecting, from a range of list of parameters values, one or more values for each region which is indicative of the image quality in the region. For instance, the image quality in the central region 301 may be assumed to be highest, and to be equal to a default value which does not need to be send by the NED 100 to the computing device, and the image quality for the one or more further regions, such as peripheral region 302, may be indicated as a relative decrease in image quality compared to the central region 301. As an example, a value of “0.5” or “2” may be used to indicate that the image quality on the peripheral region 302 is half of that in the central region 301. As an alternative, the NED 100 may select values for one or more video coding parameters, such was qp values, Adaptive Loop Filter (ALF) parameters, Luma Mapping with Chroma Scaling (LMCS) parameters, scaling list parameters, or other parameters which have an impact on the perceived image quality and the size or bitrate of the encoded image frames.

[0052] In selecting the at least two regions of the image frames and / or the image quality for the at least two regions of the image frames, either by the NED 100 or by the computing device 200, as is described further below, embodiments of the invention may be operative to select only one of the at least two regions of the image frames and the image quality for the at least two regions of the image frames, or both. For instance, if a static partitioning of the entire area 300 of an image frame into at least two regions is used, such as the central region 301 and the peripheral region 302, embodiments of the invention may be operative to only selected the relative image quality for the peripheral region 302, relative to that of the centra region 301 . Alternatively, embodiments of the invention may rely on a fixed number of regions, e.g., two regions such as regions 301 and 302 sketched in Fig.3, and static (relative) image quality values for the two regions. As an example, embodiments of the invention may be operative to use a highest image quality for the central region 301 and a lower image quality for the peripheral region 302, e.g., half of that of the central region 301 . In that case, the size of the central region 301 may be selected, e.g., by selecting a radius or other parameter defining the size of the central region 301 , such as the number of coding blocks.

[0053] As an alternative, the NED 100 may further be operative to select 412 the at least two regions of the image frames and / or the image quality for the at least two regions of the image frames, and to send 414 information pertaining to the at least two regions and / or the image quality for the at least two regions to the computing device 200. The at least two regions of the image frames, and / or the image quality of the for the at least two regions, are selected 412 based on information pertaining to a spatial variation of the effective display quality over an FoV of the NED 100. Similar to what is described hereinbefore, the information pertaining to the spatial variation of the effective display quality over the FoV of the NED 100 may be in the form of a map, a mathematical function, or other representation. The information may be stored in the memory 105 of the NED 100, e.g., during manufacturing, or accessible to the NED 100 by querying a repository.

[0054] The NED 100 may further be operative to determine 415 a point-of- gaze of an eye 110 of a user of the NED 100 on the display element 101 , and to send 416 information pertaining to the point-of-gaze to the computing device 200. The point-of-gaze of the eye 110 of the user, i.e. , the where on the display element 101 , and accordingly where on the displayed image frames, the user is gazing, may be determined 415 using a known gaze detection technology, e.g., using structured light which is reflected by the eye(s) 110. In practice, this may be achieved by using a gaze detector which is comprised in the NED 100 (not illustrated in Fig. 1 ). Advantageously, the computing device 200 may use the information about the point-of-gaze of the eye 110 of the user to select 422 the at least two regions of the image frames and / or the image quality for the at least two regions further based on the information about the point-of-gaze.

[0055] In the following, embodiments of a computing device 200 for rendering image frames for display by a NED comprising a display element with equidistant pixels, such as the NED 100 described hereinbefore, are described with reference to Fig. 2. Further reference is made to Fig. 3, which schematically illustrates an image frame divided into two regions, in accordance with embodiments of the invention, and Fig. 4, which shows a sequence diagram illustrating embodiments of the invention.

[0056] As is illustrated in Fig. 2, the computing device 200 comprises one or more processors 204, such as CPUs, microprocessors, application processors, application-specific processors, GPUs, DSPs including image processors, video codecs, or a combination thereof, and a memory 205 comprising a computer program 206, i.e., software, comprising instructions. When executed by the processor(s) 204, the instructions cause the computing device 200 to become operative in accordance with embodiments of the invention disclosed herein. The computing device 200 may alternatively or additionally comprise one or more ASICs, FPGAs, or the like, which are operative to cause the computing device 200 to become operative in accordance with embodiments of the invention disclosed herein.

[0057] More specifically, the computing device 200 is operative to encode 424 the image frames with different image quality for at least two regions of the image frames. These two regions of the image frames correspond to regions of the display element 101 having different average effective display quality across the FoV of the NED 100. In practice, this mean that the images frames are encoded 424, by the computing device 200, before transmission to the NED 100, using a video coding technology such as HEVC, WC, or the like, with different settings or parameters for the at least two regions of the image frames. The different settings or parameters relate to the image quality of the encoded image frames, and in turn to the size (in terms of bits or bytes) and bitrate of the encoded image frames. For example, the different settings or parameters may, e.g., be different resolutions, different quantization values (e.g., qp values), ALF parameters, LMCS parameters, scaling list parameters, or the like. Typically, a higher image quality results in a higher size and bitrate of the encoded image frames 425, whereas a lower image quality results in a lower size and bitrate of the encoded image frames 425. In this way, the need for a high image quality in regions of the image frames which the user can perceive by virtue of a high effective display quality, can be balanced with the need to reduce the bandwidth required for transmitting 425 the encoded image frames from the computing device 200 to the NED 100.

[0058] Optionally, the at least two regions of the image frames may comprise a central region 301 having the highest image quality, as is described hereinbefore with reference to Fig. 3. Encoding 424 image frames with different image quality in at least two regions of the image frames may be accomplished using known video coding technologies like HEVC or WC, relying on the concept of blocks, also referred to as macroblocks. Depending on the respective sizes of the at least two regions of the image frames, each region of the image frames may comprise one or more blocks, tiles, or slices, which are known from HEVC and VVC. More specifically, HEVC and WC allow recursive partitioning of an image frame into square blocks (aka coding tree units, CTUs). One CTU consists of a block of luma samples together with two corresponding blocks of chroma samples. The maximum permitted block size in VVC 128 x 128 pixels. A tile is a sequence of CTUs that covers the rectangular region of a picture, and a slice consists of a whole number of complete tiles, or of a whole number of consecutive and complete CTU rows within a tile of an image frame.

[0059] The partitioning of the regions of an image frame into blocks is exemplified in Fig.3, which shows the central circular region 301 approximated as a collection of square blocks of the same size (shown with dotted lines). Correspondingly, the peripheral region 302 is a collection of blocks, either of the same or different sizes (not illustrated in Fig. 3). In embodiments of the invention, the blocks in the central region 301 and the blocks in the peripheral region 302 are encoded with different settings or parameters, as described hereinbefore, resulting in different image qualities of the blocks in the different regions, and accordingly in different image qualities in the different regions.

[0060] The computing device 200 is further operative to send the encoded image frames to the NED 100. For example, the computing device 200 may be, operative to send the encoded image frames to the NED via a wireless connection. Such a wireless connection may commence between communications interface circuitry 207 comprised in the computing device 200, and corresponding communications interface circuitry 107 comprised in the NED 100. The communications interface circuitry 207 may, e.g., be a cellular modem, a Bluetooth modem, a Wi-Fi / WLAN modem, an Ethernet modem, or the like, and operative to exchange date, in particular a stream of encoded image frames and supplementary information, as described herein, between the NED 100 and the computing device 200.

[0061] The computing device 200 may further be operative to acquire 411 / 441 information pertaining to a spatial variation of the effective display quality over an FoV of the NED 100. The computing device 200 may be operative to acquire such information by receiving 411 the information from the NED 100, e.g., during a handshake or pairing procedure. Alternatively, the computing device 200 may be operative to acquire 441 such information from a repository 400, e.g., using a URL or other link pointing to the repository 400, a model identifier, an identifier of the NED 100, or the like. The information pertaining to a spatial variation of the effective display quality may be represented by a map, or represented by a mathematical function, e.g., as a function of radial distance from the optical axis 120. The information may either pertain to a quantified display quality, e.g., sharpness, or to a relative display quality, e.g., indicating a change in display quality or coding efficiency relative to a central region having the highest quality. The information may be generic to a type or model of the NED 100, or specific, e.g., obtained by measuring the effective display quality of the NED 100 as part of a calibration procedure. If information pertaining to a spatial variation of the effective display quality of the NED 100 is not available to the computing device, a default set of at least two regions and the corresponding image quality may be utilized.

[0062] The computing device 200 may further be operative to select 422 the at least two regions of the image frames and / or the image quality for the at least two regions of the image frames based on the acquired information pertaining to a spatial variation of the effective display quality over an FoV of the NED 100. Alternatively, the computing device 200 may further be operative to receive 414 information pertaining to the at least two regions of the image frames and / or the image quality for the at least two regions from the NED 100. In this case, the at least two regions of the image frames and / or the image quality for the at least two regions are selected 412 by the NED 100.

[0063] The computing device 200 may further be operative to receive 416 information pertaining to a point-of-gaze of an eye 110 of a user of the NED 100 on the display element, and to select 422 the at least two regions of the image frames and / or the image quality for the at least two regions further based on the received information pertaining to a point-of-gaze. Optionally, the computing device 200 may be operative to encode 424 the image frames with a lower, i.e. , decreased, image quality in a region of the at least two regions if the point-of-gaze is outside the region. For example, if the point-of- gaze is not in the central region 301 , the image frames may be encoded 424 with a lower image quality because the user is not gazing at the central region 301 . Typically, the central region 301 of the image frames may be encoded 424 with a high image quality because the effective display quality is highest in the central region 301 , close to the optical axis 120. Lowering the image quality of the encoded image frames in the central region 301 during times when the user is not gazing at the central region is advantageous in that the image frames can be encoded 424 with lower image quality. Thereby, the required bandwidth for sending 425 the encoded image frames over the (wireless) connection between the computing device 200 and the NED 100 is reduced.

[0064] In the following, embodiments of a method 500 performed by a NED comprising a display element with equidistant pixels are described with reference to Fig. 5.

[0065] The method 500 comprises receiving 506 encoded image frames from a computing device 200. The received image frames having different image quality for at least two regions 301 and 302 of the image frames. The at least two regions 301 and 302 correspond to regions of an FoV of the NED 100 which have different average effective display quality. The method 500 further comprises decoding 507 the encoded image frames, and displaying 508 the decoded image frames using the display element 101.

[0066] The NED 100 may further comprise a positive lens 102 at a distance to the display element 101 which is smaller than a focal length of the lens 102.

[0067] The effective display quality may decrease with distance from an optical axis 120 of the NED 100.

[0068] The effective display quality may be one or more of: sharpness, contrast, and resolution.

[0069] The method 500 may further comprise sending 501 information pertaining to a spatial variation of the effective display quality over the FoV of the NED 100 to the computing device 200.

[0070] The method 500 may further comprise selecting 502 the at least two regions 301 and 302 of the image frames and / or the image quality for the at least two regions 301 , 302 of the image frames based on information pertaining to a spatial variation of the effective display quality over the FoV of the NED 100. The method 500 may further comprise sending 503 information pertaining to the at least two regions 301 and 302 and / or the image quality for the at least two regions 301 and 302 to the computing device 200.

[0071] The method 500 may further comprises determining 504 a point-of-gaze of an eye 110 of a user of the NED 100 on the display element 101 . The method 500 may further comprise sending 505 information pertaining to the point-of-gaze to the computing device 200.

[0072] The at least two regions 301 and 302 of the image frames may comprise a central region 301 having the highest image quality.

[0073] The encoded image frames may be received 506 from the computing device 200 via a wireless connection.

[0074] It will be appreciated that the method 500 may comprise additional, alternative, or modified, steps in accordance with what is described throughout this disclosure. An embodiment of the method 500 may be implemented as the computer program 106 comprising instructions which, when the computer program 106 is executed by one or more processor(s) 104 comprised in the NED 100, cause the NED 100 to carry out the method 500 and become operative in accordance with embodiments of the invention described herein. The computer program 106 may be stored in a computer-readable data carrier, such as the memory 105. Alternatively, the computer program 106 may be carried by a data carrier signal, e.g., downloaded to the memory 105 via the communications interface circuitry 107.

[0075] In the following, embodiments of a method 600 of rendering image frames for display by a NED 100 comprising a display element 101 with equidistant pixels are described with reference to Fig. 6.

[0076] The method 600 comprises encoding 605 the image frames with different image quality for at least two regions 301 and 302 of the image frames. The method 600 further comprises sending 606 the encoded image frames to the NED 100.

[0077] The at least two regions 301 and 302 of the image frames may correspond to regions of an FoV of the NED 100 having different average effective display quality.

[0078] The method 600 may further comprise acquiring 601 information pertaining to a spatial variation of the effective display quality over the FOV of the NED 100.

[0079] The method 600 may further comprise selecting 603 the at least two regions 301 and 302 of the image frames and / or the image quality for the at least two regions 301 and 302 of the image frames based on the acquired information pertaining to a spatial variation of the effective display quality over the FoV of the NED 100. The method 600 may further comprise receiving 604 information pertaining to the at least two regions 301 and 302 of the image frames and / or the image quality for the at least two regions 301 and 302 from the NED 100.

[0080] The at least two regions 301 and 302 of the image frames may comprise a central region 301 having the highest image quality.

[0081] The method 600 may further comprise receiving 602 information pertaining to a point-of-gaze of an eye 110 of a user of the NED 100 on the display element 101 . The method 600 may further comprise selecting 603 the at least two regions 301 and 302 of the image frames and / or the image quality for the at least two regions 301 and 302 further based on the received information pertaining to a point-of-gaze.

[0082] The image frames may be encoded with a decreased image quality in a region of the at least two regions 301 and 302 if the point-of-gaze is outside the region.

[0083] The encoded image frames may be sent 606 to the NED 100 via a wireless connection.

[0084] It will be appreciated that the method 600 may comprise additional, alternative, or modified, steps in accordance with what is described throughout this disclosure. An embodiment of the method 600 may be implemented as the computer program 206 comprising instructions which, when the computer program 206 is executed by one or more processor(s) 204 comprised in the computing device 200, cause the computing device 200 to carry out the method 600 and become operative in accordance with embodiments of the invention described herein. The computer program 206 may be stored in a computer-readable data carrier, such as the memory 205. Alternatively, the computer program 206 may be carried by a data carrier signal, e.g., downloaded to the memory 205 via the communications interface circuitry 207.

[0085] The person skilled in the art realizes that the invention by no means is limited to the embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims.

Claims

24CLAIMS1. A Near-Eye Display, NED, (100) comprising a display element (101 ) with equidistant pixels, the NED (100) operative to: receive (425) encoded image frames from a computing device (200), the received image frames having different image quality for at least two regions (301 , 302) of the image frames corresponding to regions of a Field- of-View, FoV, of the NED (100) having different average effective display quality, decode (417) the encoded image frames, and display (418) the decoded image frames using the display element (101 ).

2. The NED (100) according to claim 1 , further comprising a positive lens (102) at a distance to the display element (101 ) which is smaller than a focal length of the lens (102).

3. The NED (100) according to claim 1 or 2, wherein the effective display quality decreases with distance from an optical axis (120) of the NED (100).

4. The NED (100) according to any one of claims 1 to 3, wherein the effective display quality is one or more of: sharpness, contrast, and resolution.

5. The NED (100) according to any one of claims 1 to 4, further operative to send (411 ) information pertaining to a spatial variation of the effective display quality over the FoV of the NED (100) to the computing device (200).

6. The NED (100) according to any one of claims 1 to 4, further operative to: select (412) the at least two regions (301 , 302) of the image frames and / or the image quality for the at least two regions (301 , 302) of the image frames based on information pertaining to a spatial variation of the effective display quality over the FoV of the NED (100), and send (414) information pertaining to the at least two regions (301 , 302) and / or the image quality for the at least two regions (301 , 302) to the computing device (200).

7. The NED (100) according to any one of claims 1 to 6, further operative to: determine (415) a point-of-gaze of an eye (110) of a user of the NED (100) on the display element (101 ), and send (416) information pertaining to the point-of-gaze to the computing device (200).

8. The NED (100) according to any one of claims 1 to 7, the at least two regions (301 , 302) of the image frames comprising a central region (301 ) having the highest image quality.

9. The NED (100) according to any one of claims 1 to 8, each of the at least two regions (301 , 302) of the image frames comprising one or more blocks, tiles, or slices.

10. The NED (100) according to any one of claims 1 to 9, operative to receive (425) the encoded image frames from the computing device (200) via a wireless connection.11 . A computing device (200) for rendering image frames for display by a Near-Eye Display, NED, (100) comprising a display element (101 ) with equidistant pixels, the computing device (200) operative to: encode (424) the image frames with different image quality for at least two regions (301 , 302) of the image frames, and send (425) the encoded image frames to the NED (100).

12. The computing device (200) according to claim 11 , the at least two regions (301 , 302) of the image frames corresponding to regions of a Field- of-View, FoV, of the NED (100) having different average effective display quality.

13. The computing device (200) according to claim 11 or 12, further operative to acquire (411 , 441 ) information pertaining to a spatial variation of the effective display quality over the FoV of the NED (100).

14. The computing device (200) according to claim 13, further operative to select (422) the at least two regions (301 , 302) of the image frames and / or the image quality for the at least two regions (301 , 302) of the image frames based on the acquired information pertaining to a spatial variation of the effective display quality over the FoV of the NED (100).

15. The computing device (200) according to claim 11 or 12, further operative to receive (414) information pertaining to the at least two regions (301 , 302) of the image frames and / or the image quality for the at least two regions (301 , 302) from the NED (100).

16. The computing device (200) according to any one of claims 11 to 15, the at least two regions (301 , 302) of the image frames comprising a central region (301 ) having the highest image quality.2717. The computing device (200) according to any one of claims 11 to 16, further operative to: receive (416) information pertaining to a point-of-gaze of an eye (110) of a user of the NED (100) on the display element (101 ), and select (422) the at least two regions (301 , 302) of the image frames and / or the image quality for the at least two regions (301 , 302) further based on the received information pertaining to a point-of-gaze.

18. The computing device (200) according to claim 17, operative to encode (424) image frames with a decreased image quality in a region of the at least two regions (301 , 302) if the point-of-gaze is outside the region.

19. The computing device (200) according to any one of claims 11 to 18, each of the at least two regions (301 , 302) of the image frames comprising one or more blocks, tiles, or slices.

20. The computing device (200) according to any one of claims 11 to 19, operative to send (425) the encoded image frames to the NED (100) via a wireless connection.

21. A method (500) performed by a Near-Eye Display, NED, (100) comprising a display element (101 ) with equidistant pixels, the method comprising: receiving (506) encoded image frames from a computing device (200), the received image frames having different image quality for at least two regions (301 , 302) of the image frames corresponding to regions of a Field- of-View, FoV,of the NED (100) having different average effective display quality, decoding (507) the encoded image frames, and28 displaying (508) the decoded image frames using the display element (101 ).

22. The method (500) according to claim 21 , the NED (100) further comprising a positive lens (102) at a distance to the display element (101 ) which is smaller than a focal length of the lens (102).

23. The method (500) according to claim 21 or 22, wherein the effective display quality decreases with distance from an optical axis (120) of the NED (100).

24. The method (500) according to any one of claims 21 to 23, wherein the effective display quality is one or more of: sharpness, contrast, and resolution.

25. The method (500) according to any one of claims 21 to 24, further comprising sending (501 ) information pertaining to a spatial variation of the effective display quality over the FoV of the NED (100) to the computing device (200).

26. The method (500) according to any one of claims 21 to 24, further comprising: selecting (502) the at least two regions (301 , 302) of the image frames and / or the image quality for the at least two regions (301 , 302) of the image frames based on information pertaining to a spatial variation of the effective display quality over the FoV of the NED (100), and sending (503) information pertaining to the at least two regions (301 , 302) and / or the image quality for the at least two regions (301 , 302) to the computing device (200).2927. The method (500) according to any one of claims 21 to 26, further comprising: determining (504) a point-of-gaze of an eye (110) of a user of the NED (100) on the display element (101 ), and sending (505) information pertaining to the point-of-gaze to the computing device (200).

28. The method (500) according to any one of claims 21 to 27, the at least two regions (301 , 302) of the image frames comprising a central region (301 ) having the highest image quality.

29. The method (500) according to any one of claims 21 to 28, each of the at least two regions (301 , 302) of the image frames comprising one or more blocks, tiles, or slices.

30. The method (500) according to any one of claims 21 to 29, wherein the encoded image frames are received (506) from the computing device (200) via a wireless connection.31 . A method (600) of rendering image frames for display by a NearEye Display, NED, (100) comprising a display element (101 ) with equidistant pixels, the method performed by a computing device (200) and comprising: encoding (605) the image frames with different image quality for at least two regions (301 , 302) of the image frames, and sending (606) the encoded image frames to the NED (100).

32. The method (600) according to claim 31 , the at least two regions (301 , 302) of the image frames corresponding to regions of a Field- of-View, FoV, of the NED (100) having different average effective display quality.3033. The method (600) according to claim 31 or 32, further comprising acquiring (601) information pertaining to a spatial variation of the effective display quality over the FOV of the NED (100).

34. The method (600) according to claim 33, further comprising selecting (603) the at least two regions (301 , 302) of the image frames and / or the image quality for the at least two regions (301 , 302) of the image frames based on the acquired information pertaining to a spatial variation of the effective display quality over the FoV of the NED (100).

35. The method (600) according to claim 31 or 32, further comprising receiving (604) information pertaining to the at least two regions (301 , 302) of the image frames and / or the image quality for the at least two regions (301 , 302) from the NED (100).

36. The method (600) according to any one of claims 31 to 35, the at least two regions (301 , 302) of the image frames comprising a central region (301 ) having the highest image quality.

37. The method (600) according to any one of claims 31 to 36, further comprising: receiving (602) information pertaining to a point-of-gaze of an eye (110) of a user of the NED (100) on the display element (101 ), and selecting (603) the at least two regions (301 , 302) of the image frames and / or the image quality for the at least two regions (301 , 302) further based on the received information pertaining to a point-of-gaze.3138. The method (600) according to claim 37, wherein the image frames are encoded with a decreased image quality in a region of the at least two (301 , 302) regions if the point-of-gaze is outside the region.

39. The method (600) according to any one of claims 31 to 38, each of the at least two regions (301 , 302) of the image frames comprising one or more blocks, tiles, or slices.

40. The method (600) according to any one of claims 31 to 39, wherein the encoded image frames are sent (606) to the NED (100) via a wireless connection.

41. A computer program (106) comprising instructions which, when the computer program (106) is executed by one or more processors (104) comprised in a Near-Eye Display, NED, (100), cause the NED (100) to carry out the method (500) according to any one of claims 21 to 30.

42. A computer program (206) comprising instructions which, when the computer program (206) is executed by one or more processors (204) comprised in a computing device (200), cause the computing device (200) to carry out the method (600) according to any one of claims 31 to 40.

Images