Method to manage scene rendering based on the confidence of the pose prediction

Abstract

Some embodiments of a method may include: obtaining a first predicted frame display time; obtaining first predicted pose information representing a prediction of a user pose at the first predicted frame display time; determining first pose confidence information indicating a confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; and causing display of the selected frame. When the pose confidence information indicates a sufficiently high confidence, a newly rendered frame is obtained for display. When the pose confidence information indicates a low confidence, a previously rendered frame is used for display, possibly after reprojection.

Description

METHOD TO MANAGE SCENE RENDERING BASED ON THE CONFIDENCE OF THE POSE PREDICTIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims benefit of European Patent Application No. EP23306698, entitled “METHOD TO MANAGE SCENE RENDERING BASED ON THE CONFIDENCE OF THE POSE PREDICTION” and filed October 3, 2023; and European Patent Application No. EP23306353, entitled “METHOD TO MANAGE SCENE RENDERING BASED ON THE CONFIDENCE OF THE POSE PREDICTION” and filed August 9, 2023, which are hereby incorporated by reference in their entirety.BACKGROUND

[0002] In an augmented reality (AR), extended reality (XR), virtual reality (VR) and / or mixed reality (MR) experience, computer-generated virtual elements are displayed to a user, e.g. in the user’s real environment or in a virtual environment using various equipment such as VR headsets, optical see- through glasses or video see-through devices (e.g. smartphone, tablet, headset).

[0003] Pose prediction is used to compensate for the round-trip time needed to render the virtual scene. For each view, the pose prediction allows the adjustment of the final rendering by taking into account the movements of the AR Device during rendering computation.

[0004] For each view, pose information can be predicated at the beginning of the rendering loop based on the previous / past pose values for a given time. Pose information may be gather, for example, using embedded device sensors such as depth or color cameras and an Inertial Measurement Unit (IMU) for the user pose estimation.SUMMARY

[0005] Embodiments described herein include methods that are used in video encoding and decoding (collectively “coding”).

[0006] A first example method in accordance with some embodiments may include: obtaining a first predicted frame display time; obtaining first predicted pose information representing a prediction of a user pose at the first predicted frame display time; obtaining first pose confidence information indicating a confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; and causing display of the selected frame.

[0007] Some embodiments of the first example method may further include: obtaining a second predicted frame display time; and performing reprojection of the selected frame based on the second predicted frame display time before causing display of the selected frame.

[0008] For some embodiments of the first example method, in response to a determination that the first pose confidence information indicates a confidence level at least as great as a threshold, a newly- rendered frame is selected as the selected frame to be displayed.

[0009] Some embodiments of the first example method may further include rendering the selected frame based on the first predicted pose.

[0010] For some embodiments of the first example method, in response to a determination that the first pose confidence information indicates a confidence level below a threshold, a previously-rendered frame is selected as the selected frame to be displayed.

[0011] For some embodiments of the first example method, the first pose confidence information comprises a plurality of flags.

[0012] For some embodiments of the first example method, the first pose confidence information comprises at least a position validity flag and an orientation validity flag.

[0013] For some embodiments of the first example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame may be selected as the selected frame to be displayed.

[0014] For some embodiments of the first example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously- rendered frame may be selected as the selected frame to be displayed.

[0015] For some embodiments of the first example method, the first pose confidence information may include at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

[0016] Some embodiments of the first example method may further include: obtaining second predicted pose information representing a prediction of a user pose at a display time of the selected frame; and obtaining second pose confidence information indicating a confidence level of the second predicted pose information.

[0017] Some embodiments of the first example method may further include calculating a pose error based on a difference between the first predicted pose information and the second predicted pose information.

[0018] Some embodiments of the first example method may further include determining, based on the first pose confidence information and the second pose confidence information, whether to calculate the pose error, the pose error being calculated in response to a determination to calculate the pose error.

[0019] Some embodiments of the first example method may further include determining, based on the first pose confidence information and the second pose confidence information, whether the pose error is valid.

[0020] For some embodiments of the first example method, obtaining the first pose confidence information comprises determining the first pose confidence information.

[0021] A first example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0022] A second example method in accordance with some embodiments may include: obtaining a first predicted frame display time; obtaining first predicted pose information representing a prediction of a user pose at the first predicted frame display time; determining first pose confidence information indicating a confidence level of the first predicted pose information; communicating, to an edge application server, the determined first pose confidence information indicating the confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; and causing display of the selected frame.

[0023] Some embodiments of the second example method may further include: obtaining a second predicted frame display time; and performing reprojection of the selected frame based on the second predicted frame display time before causing display of the selected frame.

[0024] For some embodiments of the second example method, the first pose confidence information may include a plurality of flags.

[0025] For some embodiments of the second example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag.

[0026] For some embodiments of the second example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to adetermination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

[0027] For some embodiments of the second example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously-rendered frame is selected as the selected frame to be displayed.

[0028] For some embodiments of the second example method, the first pose confidence information may include at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

[0029] Some embodiments of the second example method may further include: obtaining second predicted pose information representing a prediction of a user pose at a display time of the selected frame; and obtaining second pose confidence information indicating a confidence level of the second predicted pose information.

[0030] Some embodiments of the second example method may further include calculating a pose error based on a difference between the first predicted pose information and the second predicted pose information.

[0031] Some embodiments of the second example method may further include determining, based on the first pose confidence information and the second pose confidence information, whether to calculate the pose error, the pose error being calculated in response to a determination to calculate the pose error.

[0032] Some embodiments of the second example method may further include determining, based on the first pose confidence information and the second pose confidence information, whether the pose error is valid.

[0033] A second example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0034] A third example method in accordance with some embodiments may include: receiving a first predicted frame display time; receiving first predicted pose information representing a prediction of a user pose at the first predicted frame display time; receiving first pose confidence information indicating a confidence level of the first predicted pose information; selecting a frame for display based at least inpart on the first pose confidence information; rendering the selected frame; and sending, to an extended reality (XR) application device, the rendered frame.

[0035] For some embodiments of the third example method, in response to a determination that the first pose confidence information indicates a confidence level at least as great as a threshold, a newly- rendered frame is selected as the selected frame to be displayed.

[0036] For some embodiments of the third example method, rendering the selected frame is based on the first predicted pose.

[0037] For some embodiments of the third example method, in response to a determination that the first pose confidence information indicates a confidence level below a threshold, a previously-rendered frame is selected as the selected frame to be displayed.

[0038] For some embodiments of the third example method, the first pose confidence information may include a plurality of flags.

[0039] For some embodiments of the third example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag.

[0040] For some embodiments of the third example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

[0041] For some embodiments of the third example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously-rendered frame is selected as the selected frame to be displayed.

[0042] For some embodiments of the third example method, the first pose confidence information may include at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

[0043] Some embodiments of the third example method may further include sending, to the extended reality (XR) application device, the first predicted frame display time.

[0044] A third example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0045] A fourth example method in accordance with some embodiments may include: obtaining a first predicted frame display time; obtaining first predicted pose information representing a prediction of a user pose at the first predicted frame display time; communicating, to an edge application server, the first predicted frame display time, the first predicted pose information, and information indicating an extended reality (XR) view state; obtaining first pose confidence information indicating a confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; and causing display of the selected frame.

[0046] Some embodiments of the fourth example method may further include: obtaining a second predicted frame display time; and performing reprojection of the selected frame based on the second predicted frame display time before causing display of the selected frame.

[0047] For some embodiments of the fourth example method, the first pose confidence information may include a plurality of flags.

[0048] For some embodiments of the fourth example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag.

[0049] For some embodiments of the fourth example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

[0050] For some embodiments of the fourth example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously-rendered frame is selected as the selected frame to be displayed.

[0051] For some embodiments of the fourth example method, the first pose confidence information may include at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

[0052] Some embodiments of the fourth example method may further include: obtaining second predicted pose information representing a prediction of a user pose at a display time of the selected frame; and obtaining second pose confidence information indicating a confidence level of the second predicted pose information.

[0053] Some embodiments of the fourth example method may further include calculating a pose error based on a difference between the first predicted pose information and the second predicted pose information.

[0054] Some embodiments of the fourth example method may further include determining, based on the first pose confidence information and the second pose confidence information, whether to calculate the pose error, the pose error being calculated in response to a determination to calculate the pose error.

[0055] Some embodiments of the fourth example method may further include determining, based on the first pose confidence information and the second pose confidence information, whether the pose error is valid.

[0056] A fourth example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0057] A fifth example method in accordance with some embodiments may include: receiving a first predicted frame display time; receiving first predicted pose information representing a prediction of a user pose at the first predicted frame display time; receiving information indicating an extended reality (XR) view state; determining first pose confidence information indicating a confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; rendering the selected frame; and sending, to an extended reality (XR) application device, the rendered frame, and the determined first pose confidence information.

[0058] For some embodiments of the firth example method, in response to a determination that the first pose confidence information indicates a confidence level at least as great as a threshold, a newly- rendered frame is selected as the selected frame to be displayed.

[0059] For some embodiments of the firth example method, rendering the selected frame is based on the first predicted pose.

[0060] For some embodiments of the firth example method, in response to a determination that the first pose confidence information indicates a confidence level below a threshold, a previously-rendered frame is selected as the selected frame to be displayed.

[0061] For some embodiments of the firth example method, the first pose confidence information may include a plurality of flags.

[0062] For some embodiments of the firth example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag.

[0063] For some embodiments of the firth example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

[0064] For some embodiments of the firth example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously- rendered frame is selected as the selected frame to be displayed.

[0065] For some embodiments of the firth example method, the first pose confidence information may include at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

[0066] Some embodiments of the fifth example method may further include sending, to the extended reality (XR) application device, the first predicted frame display time.

[0067] A fifth example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0068] A sixth example method / apparatus in accordance with some embodiments may include one or more processors configured to perform any one of the methods listed above.

[0069] A seventh example method / apparatus in accordance with some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any one of the methods listed above.

[0070] An eighth example method / apparatus in accordance with some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any one of the methods listed above.

[0071] An example computer-readable medium in accordance with some embodiments may include instructions for causing one or more processors to perform any one of the methods listed above.

[0072] For some embodiments of the example computer-readable medium, the computer-readable medium is a non-transitory storage medium.

[0073] An example computer program in accordance with some embodiments may include instructions which, when the program is executed by one or more processors, causes the one or more processors to carry out any one of the methods listed above.

[0074] An example signal in accordance with some embodiments may include a scene description file generated according to any one of the methods listed above.

[0075] In additional embodiments, encoder and decoder apparatus are provided to perform the methods described herein. An encoder or decoder apparatus may include a processor configured to perform the methods described herein. The apparatus may include a computer-readable medium (e.g. a non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, a computer-readable medium (e.g. a non-transitory medium) stores a video encoded using any of the methods described herein.

[0076] One or more of the present embodiments also provide a computer readable storage medium having stored thereon instructions for performing bi-directional optical flow, encoding or decoding video data according to any of the methods described above. The present embodiments also provide a computer readable storage medium having stored thereon a bitstream generated according to the methods described above. The present embodiments also provide a method and apparatus for transmitting the bitstream generated according to the methods described above. The present embodiments also provide a computer program product including instructions for performing any of the methods described.BRIEF DESCRIPTION OF THE DRAWINGS

[0077] FIG. 1 A is a schematic side view illustrating an example waveguide display that may be used with extended reality (XR) applications according to some embodiments.

[0078] FIG. 1 B is a schematic side view illustrating an example alternative display type that may be used with extended reality applications according to some embodiments.

[0079] FIG. 1 C is a schematic side view illustrating an example alternative display type that may be used with extended reality applications according to some embodiments.

[0080] FIG. 1 D is a system diagram illustrating an example set of interfaces for a system according to some embodiments.

[0081] FIG. 1 E is a system diagram illustrating an example communications system according to some embodiments.

[0082] FIG. 2 is a message sequencing diagram illustrating an example process for measuring the pose error and time error in pose prediction.

[0083] FIG. 3 is a schematic timing diagram illustrating an example process using a second prediction (T2.predicted2) of the display time to improve prediction accuracy.

[0084] FIG. 4 is a message sequencing diagram illustrating an example process using pose confidence information with the confidence level computed and checked by the XR application according to some embodiments.

[0085] FIG. 5 is a message sequencing diagram illustrating an example process using pose confidence information with the confidence level checked by the edge application server according to some embodiments.

[0086] FIG. 6 is a message sequencing diagram illustrating an example process using pose confidence information with the confidence level computed and checked by the edge application server according to some embodiments.

[0087] FIG. 7 is a flowchart illustrating an example process including frame selection using pose confidence information according to some embodiments.

[0088] FIG. 8 is a flowchart illustrating an example process including frame selection using pose confidence information according to some embodiments.

[0089] FIG. 9 is a flowchart illustrating an example process including a pose error determination using pose confidence information according to some embodiments.

[0090] FIG. 10 is a flowchart illustrating an example process for rendering a scene based on a confidence level of a predicted pose according to some embodiments.

[0091] FIG. 11 is a flowchart illustrating an example process for rendering a scene based on a confidence level of a predicted pose according to some embodiments.

[0092] FIG. 12 is a flowchart illustrating an example process for rendering a scene based on a confidence level of a predicted pose according to some embodiments.

[0093] FIG. 13 is a flowchart illustrating an example process for rendering a scene based on a confidence level of a predicted pose according to some embodiments.

[0094] FIG. 14 is a flowchart illustrating an example process for rendering a scene based on a confidence level of a predicted pose according to some embodiments.

[0095] The entities, connections, arrangements, and the like that are depicted in— and described in connection with— the various figures are presented by way of example and not by way of limitation. As such, any and all statements or other indications as to what a particular figure “depicts,” what a particularelement or entity in a particular figure “is” or “has,” and any and all similar statements— that may in isolation and out of context be read as absolute and therefore limiting— may only properly be read as being constructively preceded by a clause such as “In at least one embodiment, ....” For brevity and clarity of presentation, this implied leading clause is not repeated ad nauseum in the detailed description.DETAILED DESCRIPTION

[0096] FIG. 1 A is a schematic side view illustrating an example waveguide display that may be used with extended reality (XR) applications according to some embodiments. An image is projected by an image generator 102. The image generator 102 may use one or more of various techniques for projecting an image. For example, the image generator 102 may be a laser beam scanning (LBS) projector, a liquid crystal display (LCD), a light-emitting diode (LED) display (including an organic LED (OLED) or micro LED ( LED) display), a digital light processor (DLP), a liquid crystal on silicon (LCoS) display, or other type of image generator or light engine.

[0097] Light representing an image 112 generated by the image generator 102 is coupled into a waveguide 104 by a diffractive in-coupler 188. The in-coupler 188 diffracts the light representing the image 112 into one or more diffractive orders. For example, light ray 108, which is one of the light rays representing a portion of the bottom of the image, is diffracted by the in-coupler 188, and one of the diffracted orders 110 (e.g. the second order) is at an angle that is capable of being propagated through the waveguide 104 by total internal reflection. The image generator 102 displays images as directed by a control module 124, which operates to render image data, video data, point cloud data, or other displayable data.

[0098] At least a portion of the light 110 that has been coupled into the waveguide 104 by the diffractive in-coupler 188 is coupled out of the waveguide by a diffractive out-coupler 114. At least some of the light coupled out of the waveguide 104 replicates the incident angle of light coupled into the waveguide. For example, in the illustration, out-coupled light rays 116a, 116b, and 116c replicate the angle of the in-coupled light ray 108. Because light exiting the out-coupler replicates the directions of light that entered the in-coupler, the waveguide substantially replicates the original image 112. A user’s eye 118 can focus on the replicated image.

[0099] In the example of FIG. 1 A, the out-coupler 114 out-couples only a portion of the light with each reflection allowing a single input beam (such as beam 108) to generate multiple parallel output beams (such as beams 116a, 116b, and 116c). In this way, at least some of the light originating from each portion of the image is likely to reach the user’s eye even if the eye is not perfectly aligned with the center of the out-coupler. For example, if the eye 118 were to move downward, beam 116c may enter the eyeeven if beams 116a and 116b do not, so the user can still perceive the bottom of the image 112 despite the shift in position. The out-coupler 114 thus operates in part as an exit pupil expander in the vertical direction. The waveguide may also include one or more additional exit pupil expanders (not shown in FIG. 1 A) to expand the exit pupil in the horizontal direction.

[0100] In some embodiments, the waveguide 104 is at least partly transparent with respect to light originating outside the waveguide display. For example, at least some of the light 120 from real-world objects (such as object 122) traverses the waveguide 104, allowing the user to see the real-world objects while using the waveguide display. As light 120 from real-world objects also goes through the diffraction grating 114, there will be multiple diffraction orders and hence multiple images. To minimize the visibility of multiple images, it is desirable for the diffraction order zero (no deviation by 114) to have a great diffraction efficiency for light 120 and order zero, while higher diffraction orders are lower in energy. Thus, in addition to expanding and out-coupling the virtual image, the out-coupler 114 is preferably configured to let through the zero order of the real image. In such embodiments, images displayed by the waveguide display may appear to be superimposed on the real world.

[0101] FIG. 1 B is a schematic side view illustrating an example alternative display type that may be used with extended reality applications according to some embodiments. In an XR head-mounted display device 130, a control module 132 controls a display 134, which may be an LCD, to display an image. The head-mounted display includes a partly-reflective surface 136 that reflects (and in some embodiments, both reflects and focuses) the image displayed on the LCD to make the image visible to the user. The partly-reflective surface 136 also allows the passage of at least some exterior light, permitting the user to see their surroundings.

[0102] FIG. 1 C is a schematic side view illustrating an example alternative display type that may be used with extended reality applications according to some embodiments. In an XR head-mounted display device 140, a control module 142 controls a display 144, which may be an LCD, to display an image. The image is focused by one or more lenses of display optics 146 to make the image visible to the user. In the example of FIG. 1 C, exterior light does not reach the user’s eyes directly. However, in some such embodiments, an exterior camera 148 may be used to capture images of the exterior environment and display such images on the display 144 together with any virtual content that may also be displayed.

[0103] The embodiments described herein are not limited to any particular type or structure of XR display device.

[0104] FIG. 1 D is a system diagram illustrating an example set of interfaces for a system according to some embodiments. An extended reality display device, together with its control electronics, may beimplemented using a system such as the system of FIG. 1 D. System 150 can be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this document. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 150, singly or in combination, can be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 150 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 150 is configured to implement one or more of the aspects described in this document.

[0105] The system 150 includes at least one processor 152 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this document. Processor 152 may include embedded memory, input output interface, and various other circuitries as known in the art. The system 150 includes at least one memory 154 (e.g., a volatile memory device, and / or a non-volatile memory device). System 150 may include a storage device 158, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. The storage device 158 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and / or a network accessible storage device, as non-limiting examples.

[0106] System 150 includes an encoder / decoder module 156 configured, for example, to process data to provide an encoded video or decoded video, and the encoder / decoder module 156 can include its own processor and memory. The encoder / decoder module 156 represents module(s) that can be included in a device to perform the encoding and / or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 156 can be implemented as a separate element of system 150 or can be incorporated within processor 152 as a combination of hardware and software as known to those skilled in the art.

[0107] Program code to be loaded onto processor 152 or encoder / decoder 156 to perform the various aspects described in this document can be stored in storage device 158 and subsequently loaded onto memory 154 for execution by processor 152. In accordance with various embodiments, one or more ofprocessor 152, memory 154, storage device 158, and encoder / decoder module 156 can store one or more of various items during the performance of the processes described in this document. Such stored items can include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0108] In some embodiments, memory inside of the processor 152 and / or the encoder / decoder module 156 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device can be either the processor 152 or the encoder / decoder module 152) is used for one or more of these functions. The external memory can be the memory 154 and / or the storage device 158, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of, for example, a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations, such as for MPEG- 2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also referred to as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).

[0109] The input to the elements of system 150 can be provided through various input devices as indicated in block 172. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples, not shown in FIG. 1 C, include composite video.

[0110] In various embodiments, the input devices of block 172 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters,demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various of these functions, including, for example, downconverting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements can include inserting elements in between existing elements, such as, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[0111] Additionally, the USB and / or HDMI terminals can include respective interface processors for connecting system 150 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, can be implemented, for example, within a separate input processing IC or within processor 152 as necessary. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface ICs or within processor 152 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 152, and encoder / decoder 156 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.

[0112] Various elements of system 150 can be provided within an integrated housing, Within the integrated housing, the various elements can be interconnected and transmit data therebetween using suitable connection arrangement 174, for example, an internal bus as known in the art, including the I nter-IC (I2C) bus, wiring, and printed circuit boards.

[0113] The system 150 includes communication interface 160 that enables communication with other devices via communication channel 162. The communication interface 160 can include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 162. The communication interface 160 can include, but is not limited to, a modem or network card and the communication channel 162 can be implemented, for example, within a wired and / or a wireless medium.

[0114] Data is streamed, or otherwise provided, to the system 150, in various embodiments, using a wireless network such as a Wi-Fi network, for example IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal of these embodiments is received over the communications channel 162 and the communications interface 160 which are adapted for Wi-Ficommunications. The communications channel 162 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 150 using a set-top box that delivers the data over the HDMI connection of the input block 172. Still other embodiments provide streamed data to the system 150 using the RF connection of the input block 172. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth network.

[0115] The system 150 can provide an output signal to various output devices, including a display 176, speakers 178, and other peripheral devices 180. The display 176 of various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 176 can be for a television, a tablet, a laptop, a cell phone (mobile phone), or other device. The display 176 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop). The other peripheral devices 180 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 180 that provide a function based on the output of the system 150. For example, a disk player performs the function of playing the output of the system 150.

[0116] In various embodiments, control signals are communicated between the system 150 and the display 176, speakers 178, or other peripheral devices 180 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communications protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to system 150 via dedicated connections through respective interfaces 164, 166, and 168. Alternatively, the output devices can be connected to system 150 using the communications channel 162 via the communications interface 160. The display 176 and speakers 178 can be integrated in a single unit with the other components of system 150 in an electronic device such as, for example, a television. In various embodiments, the display interface 164 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0117] The display 176 and speaker 178 can alternatively be separate from one or more of the other components, for example, if the RF portion of input 172 is part of a separate set-top box. In various embodiments in which the display 176 and speakers 178 are external components, the output signalcan be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0118] The system 150 may include one or more sensor devices 168. Examples of sensor devices that may be used include one or more GPS sensors, gyroscopic sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors may be used to determine information such as user’s position and orientation. Where the system 150 is used as the control module for an extended reality display (such as control modules 124, 132), the user’s position and orientation may be used in determining how to render image data such that the user perceives the correct portion of a virtual object or virtual scene from the correct point of view. In the case of headmounted display devices, the position and orientation of the device itself may be used to determine the position and orientation of the user for the purpose of rendering virtual content. In the case of other display devices, such as a phone, a tablet, a computer monitor, or a television, other inputs may be used to determine the position and orientation of the user for the purpose of rendering content. For example, a user may select and / or adjust a desired viewpoint and / or viewing direction with the use of a touch screen, keypad or keyboard, trackball, joystick, or other input. Where the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation used for the purpose of rendering content may be selected and / or adjusted based on motion of the display device.

[0119] The embodiments can be carried out by computer software implemented by the processor 152 or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 154 can be of any type appropriate to the technical environment and can be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 152 can be of any type appropriate to the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0120] FIG. 1 E is a diagram illustrating an example communications system 182 in which one or more disclosed embodiments may be implemented. The communications system 182 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 182 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 182 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access(FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT- Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0121] As shown in FIG. 1 E, the communications system 182 may include wireless transmit / receive units (WTRUs) 184a, 184b, 184c, 184d, a RAN 186, a ON 188, a public switched telephone network (PSTN) 190, the Internet 192, and other networks 194, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 184a, 184b, 184c, 184d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 184a, 184b, 184c, 184d, any of which may be referred to as a “station” and / or a “STA”, may be configured to transmit and / or receive wireless signals and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and / or other wireless devices operating in an industrial and / or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and / or industrial wireless networks, and the like. Any of the WTRUs 184a, 184b, 184c and 184d may be interchangeably referred to as a UE.

[0122] The communications systems 182 may also include a base station 196a and / or a base station 196b. Each of the base stations 196a, 196b may be any type of device configured to wirelessly interface with at least one of the WTRUs 184a, 184b, 184c, 184d to facilitate access to one or more communication networks, such as the CN 188, the Internet 192, and / or the other networks 194. By way of example, the base stations 196a, 196b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 196a, 196b are each depicted as a single element, it will be appreciated that the base stations 196a, 196b may include any number of interconnected base stations and / or network elements.

[0123] The base station 196a may be part of the RAN 186, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 196a and / or the base station 196b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or acombination of licensed and unlicensed spectrum. A cell may provide coverage for a wireless service to a specific geographical area that may be relatively fixed or that may change over time. The cell may further be divided into cell sectors. For example, the cell associated with the base station 196a may be divided into three sectors. Thus, in one embodiment, the base station 196a may include three transceivers, i.e., one for each sector of the cell. In an embodiment, the base station 196a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in desired spatial directions.

[0124] The base stations 196a, 196b may communicate with one or more of the WTRUs 184a, 184b, 184c, 184d over an air interface 198, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 198 may be established using any suitable radio access technology (RAT).

[0125] More specifically, as noted above, the communications system 182 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 196a in the RAN 186 and the WTRUs 184a, 184b, 184c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 198 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink (DL) Packet Access (HSDPA) and / or High-Speed UL Packet Access (HSUPA).

[0126] In an embodiment, the base station 196a and the WTRUs 184a, 184b, 184c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 198 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0127] In an embodiment, the base station 196a and the WTRUs 184a, 184b, 184c may implement a radio technology such as NR Radio Access, which may establish the air interface 198 using New Radio (NR).

[0128] In an embodiment, the base station 196a and the WTRUs 184a, 184b, 184c may implement multiple radio access technologies. For example, the base station 196a and the WTRUs 184a, 184b, 184c may implement LTE radio access and NR radio access together, for instance using dual connectivity (DC) principles. Thus, the air interface utilized by WTRUs 184a, 184b, 184c may becharacterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., a eNB and a gNB).

[0129] In other embodiments, the base station 196a and the WTRUs 184a, 184b, 184c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1 X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0130] The base station 196b in FIG. 1 E may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a roadway, and the like. In one embodiment, the base station 196b and the WTRUs 184c, 184d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 196b and the WTRUs 184c, 184d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 196b and the WTRUs 184c, 184d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR etc.) to establish a picocell or femtocell. As shown in FIG. 1 E, the base station 196b may have a direct connection to the Internet 192. Thus, the base station 196b may not be required to access the Internet 192 via the CN 188.

[0131] The RAN 186 may be in communication with the CN 188, which may be any type of network configured to provide voice, data, applications, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 184a, 184b, 184c, 184d. The data may have varying quality of service (QoS) requirements, such as differing throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 188 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions, such as user authentication. Although not shown in FIG. 1 E, it will be appreciated that the RAN 186 and / or the CN 188 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 186 or a different RAT. For example, in addition to being connected to the RAN 186, which may be utilizing a NR radio technology, the CN 188 may also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0132] The CN 188 may also serve as a gateway for the WTRUs 184a, 184b, 184c, 184d to access the PSTN 190, the Internet 192, and / or the other networks 194. The PSTN 190 may include circuit- switched telephone networks that provide plain old telephone service (POTS). The Internet 192 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and / or the internet protocol (IP) in the TCP / IP internet protocol suite. The networks 194 may include wired and / or wireless communications networks owned and / or operated by other service providers. For example, the networks 194 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 186 or a different RAT.

[0133] Some or all of the WTRUs 184a, 184b, 184c, 184d in the communications system 182 may include multi-mode capabilities (e.g., the WTRUs 184a, 184b, 184c, 184d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 184c shown in FIG. 1 E may be configured to communicate with the base station 196a, which may employ a cellular-based radio technology, and with the base station 196b, which may employ an IEEE 802 radio technology.Overview of Pose Measurement and Rendering

[0134] Example embodiments are described herein with reference to the OpenXR framework, though it is contemplated that the principles described herein may be applied to rendering using other frameworks.

[0135] In example embodiments, pose prediction and tracking of an AR display device are managed by a module referred to as the XR runtime. In a runtime module using the OpenXR framework, the function xrLocateViews is called to obtain the pose value at a specific time. The function returns an associated status. For example, xrLocateViews may accept as an input a display time for which to return the resulting data. An application uses xrLocateViews to retrieve the viewer pose and projection parameters used to render each view for use in a composition projection layer. Along with the predicted pose information, the xrLocateViews function may also set values of four flags. In some embodiments, the values of the four flags or the flags themselves can be read to get specific information about the pose estimation status. Different implementations may provide status information using different techniques.

[0136] In example embodiments, the flags are used to provide information indicating a level of confidence of the pose estimation. In other embodiments, a different format may be used to provide suchinformation. In some embodiments, the confidence information is used to determine, at least in part, the process flow used for rendering of the frame, the call flow and / or the pose error measurement.

[0137] In an example embodiment (see, e.g., FIG. 4), the confidence level is computed and checked locally by the XR Application (which may be hosted by a UE (User Equipment)). In another example embodiment (see, e.g., FIG. 5), the confidence level is computed locally by the XR Application (e.g., hosted by a UE) and sent to the Edge Application Server where it is checked. In yet another example embodiment (see, e.g., FIG. 6), the view’s status flags (e.g., XrViewStateFlags flags with OpenXR) that give information validity and tracking of position and orientation are sent to the Edge Application Server which computes and checks the confidence level (rather than the XR Application doing so).

[0138] FIG. 2 is a message sequencing diagram illustrating an example process for measuring the pose error and time error in pose prediction with the confidence level computed by the XR application. A measurement procedure 200 for the scenario of cloud-based rendering is shown as an example in FIG. 2. The example of FIG. 2 does not vary the call flow based on the confidence of the pose estimation and is described here to clarify those embodiments that do make use of information regarding the confidence of pose estimation. Not all of the steps described below are necessarily performed in all embodiments. The XR Runtime and the XR Application may be on a same device such as a UE (user equipment), or on different devices such as, for example, an AR glasses (which hosts the XR Runtime) and a UE (which hosts the XR Application).

[0139] In step 1 (208), the XR Application 204 estimates the round-trip time (RTT) between the XR application 204 and the Edge Application Server (EAS) 206.

[0140] In step 2 (210), the XR Application 204 queries for the next display time. This (and step 3) can be achieved by calling the xrWaitFrame function in embodiments that use OpenXR.

[0141] In step 3 (212), the XR Runtime 202 replies with information indicating the next display time.

[0142] In step 4 (214), the XR application 204 predicts a display time - an initial prediction - and the use of initial is because a second prediction / estimation will be made later. This predicted display time is called T2.predicted1.

[0143] In step 5 (216), the XR application 204 queries for a predicted pose at the initial predicted display time T2.predicted1. Calling the function xrLocateViews in OpenXR can achieve this step and step 7.

[0144] In step 6 (218), the XR Runtime 202 predicts the pose, and the prediction occurs at time T1.

[0145] In step 7 (220), the XR Runtime 202 returns with the predicted pose (P.predicted 1 ).

[0146] In step 8 (222), the XR application 204 sends the predicted pose (P.predictedl) and the associated initial predicted display time (T2.predicted1) to the edge application server (EAS) 206 being used to perform the rendering. In some embodiments, the user equipment or other system is used to perform the rendering rather than using an EAS 206.

[0147] In step 9 (224), the EAS 206 renders for the predicted pose (P.predictedl) and compresses the rendered frame.

[0148] In step 10 (226), the EAS 206 returns the rendered frame along with the initial predicted display time (T2.predicted1) to the XR Application.

[0149] In step 11 (228), the XR Application 204 sends the rendered frame to the XR Runtime, e.g., via swapchain. This is achieved in some embodiments by calling the xrReleaseSwapchainlmage function in OpenXR. The XR Application passes the display time used for the rendering the frame, and this can be achieved by calling the xrEndFrame function in OpenXR.

[0150] In step 12 (230), the XR Application 204 queries for the predicted display time. This is intended to get a more accurate prediction of the display time than the one in step 4, because there is less time to predict into the future at this moment.

[0151] In step 13 (232), the XR Runtime 202 returns an updated prediction of the display time (T2.predicted2).

[0152] In step 14 (234), the XR Runtime 202 performs reprojection for pose correction. The actual display play time is called T2.actual.

[0153] In step 15 (236), the XR Application 204 queries for the pose associated with the updated prediction of the display time (T2. p red icted 2) . This can be achieved by calling the xrLocate Views function in OpenXR.

[0154] In step 16 (238), the XR Runtime 202 performs pose estimation.

[0155] In step 17 (240), the XR Runtime 202 returns a pose estimate (P.predicted2).

[0156] In step 18 (242), the XR Application 204 computes a pose error estimate (P.predictedl - P.predicted2) and a time error estimate (T2.predicted1 — T2.predicted2).

[0157] The User Equipment in accordance with some embodiments could, e.g., be a WTRU e.g., 184a, 184b, 184c, and 184d of FIG. 1 E. In accordance with some embodiments, User Equipment could, e.g., be implemented as, or be part of, the example system 150 of FIG. 1 D. It will be understood that these are merely examples and the User Equipment is not limited to such example implementations.

[0158] The Edge Application Server in accordance with some embodiments could, e.g., be a server that is part of or in communication with networks shown in FIG. 1 E. According to some embodiments, the edge application server may be in communication with one or more of the WTRUs (e.g., user equipment such as, e.g., head-mounted display devices or smartphones) 184a, 184b, 184c, and184d of FIG. 1 E. In accordance with some embodiments, the Edge Application Server could, e.g., be implemented as, or be part of, the example system 150 of FIG. 1 D. It will be understood that these are merely examples and the Edge Application Server is not limited to such example implementations.

[0159] FIG. 3 is a schematic timing diagram illustrating an example process using a second prediction (T2.predicted2) of the display time to improve prediction accuracy. In this example timing diagram 300, two queries are used to predict the display time of the same frame.

[0160] The first query occurs in step 2 at a first “now” time 312, and the query result is used to determine a target display time for the rendering process in step 4. In step 4, for some embodiments, the XR Application predicts an initial display time (T2. predicted 1 (306)) based on RTT 314 and the frame rate. As shown in FIG. 3, the next predicted display time 304 is equal to the (previous) display time 302 plus 1 / FrameRate (308). Also, T2.predicted1 (306) is equal to the next predicted display time 304 plus 1 / FrameRate (310).

[0161] The second query occurs at a second “now” time 320 much closer to the actual display time, as shown in steps 12-13, and thus provides higher accuracy. The relationships among the actual display times 316, 318 and the prediction T2.predicted2 (322) are illustrated in FIG. 3.

[0162] Within the OpenXR framework, the xrLocateViews function is defined as follows: / / Provided by XR_VERS ION_1_0 XrResult xrLocateViews (XrSession session, const XrViewLocatelnf o* viewLocatelnfo ,XrViewState* viewstate , uint32_t viewCapacitylnput , uint32_t* viewCountOutput ,XrView* views ) ;

[0163] The parameters used by the xrLocateViews function may be described as follows: session is a handle to the provided XrSession.• viewLocatelnfo is a pointer to a valid XrViewLocatelnfo structure, which includes information indicating an estimated display time.• viewState is the output structure with the viewer state information, which may include flags used to convey confidence information in some embodiments.• viewCapacitylnput is an input parameter which specifies the capacity of the views array.• viewCount Output is an output parameter which identifies the valid count of views.• views is an array of XrView and may be used to provide pose information.

[0164] An estimated display time is provided to the xrLocate Views function through the viewLocatelnfo parameter. In the “view” parameter of type XRView, the xrLocateViews function provides the pose information for the estimated display time. This estimated display time may be the target display time for a given frame.

[0165] The function xrLocateViews returns an array of XrView elements, one for each view of the specified view configuration type, along with an XrViewState containing additional state data shared across all views. The XrView elements may be configured as follows to include pose information (e.g. information of type XrPosef). typedef struct XrView {XrStructureType type ; void* next ;XrPosef pose ;XrFovf fov;} XrView;

[0166] The XrViewState structure contains containing additional state data as follows: typedef struct XrViewState {XrStructureType type ; void* next ;XrViewStateFlags viewStateFlags ;} XrViewState ;

[0167] The structure XrViewState contains a field XrViewStateFlags. The XrViewStateFlags field contains a bitmask of XrViewStateFlagBits indicating state for all views as follows: / / Flag bits for XrViewStateFlags static const XrViewStateFlags XR_VIEW_STATE_ORIENTATION_VALID_BIT =0x00000001 ; static const XrViewStateFlags XR_VIEW_STATE_POSITION_VALID_BIT =0x00000002 ; static const XrViewStateFlagsXR_VIEW_STATE_ORIENTATION_TRACKED_BIT=OxO O O O O O Q4 ; static const XrViewStateFlags XR_VIEW_STATE_POSITION_TRACKED_BIT =0x00000008 ;

[0168] Conventionally, these state flags are used to indicate the following information:. XR_VIEW_STATE_ORIENTATION_VALID_BIT indicates whether all XrView orientations contain valid data.• XR_VIEW_STATE_POSITION_VALID_BIT indicates whether all XrView positions contain valid data.. XR_VIEW_STATE_ORIENTATION_TRACKED_BIT indicates whether all XrView orientations represent an actively tracked orientation.. XR_VIEW_STATE_POSITION_TRACKED_BIT indicates whether all XrView positions represent an actively tracked position.

[0169] In example embodiments, these state flags are further used to indicate a level of confidence in the pose information.Signaling and Use of Confidence Information

[0170] In the example call flow as shown in FIG. 2, the function xrLocateViews is called two times (step 5 and step 15) to obtain a predicted pose.

[0171] At steps 7 and step 17, pose estimates are obtained along with information regarding the validity and tracking of position and orientation information. Such information regarding the validity and tracking may be provided in using XrViewStateFlags flags.

[0172] In example embodiments, the XR Runtime provides information indicating the confidence in the pose estimate. Such information may be used to manage the quality of experience (QOE) and the rendering. In some embodiments using the OpenXR framework, the information indicating the level of confidence in the pose estimate is provided by leveraging the existing XrViewStateFlags flags, namely:XR_VIEW_STATE_ORIENTATION_VALID_BIT• XR_VIEW_STATE_POSITION_VALID_BIT• XR_VIEW_STATE_POSITION_TRACKED_BIT• XR_VIEW_STATE_ORIENTATION_TRACKED_BIT

[0173] In OpenXR, these flags are provided as a bitmask. To get the value, a combination of the mask with XrViewStateFlagBits is performed (for example: XrViewStateFlagBits & XR_VIEW_STATE_ORIENTATION_VALID_BIT).

[0174] For simplicity, in the following, XR_VIEW_STATE_POSITION_VALID_BIT should be understood as the result of the masking operation. The same is true for the three other flags. For descriptive purposes, 0 is used for “not valid” and 1 is used to indicate “valid”.

[0175] The four state flags together are capable of distinguishing sixteen (24) different states. However, this many different states are not needed to provide the necessary information regarding validity and tracking of pose information. Thus, the conventional use of the four XrViewStateFlags includes redundant information. For example, it may be considered that VALID information may be more important than the TRACKED information. That is, TRACKED is not relevant if VALID is equal to 0; TRACKED can be equal to 0 while VALID is equal to 1 . Other conventions may alternatively be adopted to allow the four XrViewStateFlags to provide confidence level information in addition to the conventional information regarding validity and tracking.

[0176] In example embodiments, the XrViewStateFlags provided by the xrLocateViews function (e.g. in step 7) are used to signal the confidence level of the predicted pose information. As an example, confidence levels may be signaled using the flag values in XrViewStateFlags as indicated in Table 1. However, different confidence levels may be assigned to the flag combinations in different embodiments. In the example of Table 1 , the “X” indicates that, given other flag values in the same row, the flag value marked “X” does not affect the determined confidence level.Table 1

[0177] As noted above, in an example embodiment (see, e.g., FIG. 4), the confidence level is computed and checked locally by the XR Application (which may be hosted by a UE (User Equipment). In another example embodiment (see, e.g., FIG. 5), the confidence level is computed locally by the XR Application (e.g., hosted by a UE) and sent to the Edge Application Server where it is checked. In a yet another example embodiment (see, e.g., FIG. 6), the view’s status flags (e.g., XrViewStateFlags flags with OpenXR) that give information validity and tracking of position and orientation are sent to the Edge Application Server which computes and checks the confidence level (rather than the XR Application doing so). The example embodiments and categories of embodiments are merely examples of how some example processing may be shifted in varying degrees between the XR Application (e.g., hosted by a UE) and the Edge Application Server, and the embodiments described herein are not limited to these implementations.

[0178] FIG. 4 is a message sequencing diagram illustrating an example process using pose confidence information with the confidence level computed and checked by the XR application according to some embodiments. In some embodiments, the confidence level indicated by flags returned by xrLocateViews (e.g., step 7) are used to modify the rendering process of the call flow shown in FIG. 2. For example, some embodiments include a step 7bis (422) as illustrated in FIG. 4 of computing and checking for the confidence level. In some embodiments, the frame to be displayed depends on the confidence level. For example, in response to a confidence level being at or above a threshold (which may be a predetermined threshold), the scene may be rendered using the predicted pose, according to the call flow as illustrated in FIG. 2. However, in response to the confidence level being below the threshold, in some embodiments, the scene is not rendered using the predicted pose. An example of the rendering process depending on the confidence level is provided in Table 2. Steps 12 (432), 13 (434),and 14 (436) are still performed in the example embodiment of the message flow 400 shown in FIG. 4 regardless of the confidence level.

[0179] In step 1 (408), the XR Application 404 estimates the round-trip time (RTT) between the XR application 404 and the Edge Application Server (EAS) 406.

[0180] In step 2 (410), the XR Application 404 queries for the next display time. This (and step 3) can be achieved by calling the xrWaitFrame function in embodiments that use OpenXR.

[0181] In step 3 (412), the XR Runtime 402 replies with information indicating the next display time.

[0182] In step 4 (414), the XR application 404 predicts a display time - an initial prediction - and the use of initial is because a second prediction / estimation will be made later. This predicted display time is called T2.predicted1.

[0183] In step 5 (416), the XR application 404 queries for a predicted pose at the initial predicted display time T2.predicted1. Calling the function xrLocateViews in OpenXR can achieve this step and step 7.

[0184] In step 6 (418), the XR Runtime 402 predicts the pose, and the prediction occurs at time T1.

[0185] In step 7 (420), the XR Runtime 402 returns with the predicted pose (P.predictedl).

[0186] In step 7bis (422), the XR application 404 computes and checks the confidence level. If the confidence level is lower than a pre-configured threshold, for example 5 / 8, then, depending on the strategy, the XR Application 404 may take the previous rendered frame. Hence, step 8 (424), step 9 (426) and step 10 (428) are skipped. In that case, step 11 (430) is executed with the previous rendering frame.

[0187] In step 8 (424), the XR application 404 sends the predicted pose (P.predictedl) and the associated initial predicted display time (T2.predicted1) to the edge application server (EAS) 406 being used to perform the rendering. In some embodiments, the user equipment or other system is used to perform the rendering rather than using an EAS 406.

[0188] In step 9 (426), the EAS 406 renders for the predicted pose (P.predictedl) and compresses the rendered frame.

[0189] In step 10 (428), the EAS 406 returns the rendered frame along with the initial predicted display time (T2.predicted1) to the XR Application.

[0190] In step 11 (430), the XR Application 404 sends the rendered frame to the XR Runtime, e.g., via swapchain. This is achieved in some embodiments by calling the xrReleaseSwapchainlmagefunction in OpenXR. The XR Application passes the display time used for the rendering the frame, and this can be achieved by calling the xrEndFrame function in OpenXR.

[0191] In step 12 (432), the XR Application 404 queries for the predicted display time. This is intended to get a more accurate prediction of the display time than the one in step 4, because there is less time to predict into the future at this moment.

[0192] In step 13 (434), the XR Runtime 402 returns an updated prediction of the display time (T2.predicted2).

[0193] In step 14 (436), the XR Runtime 402 performs reprojection for pose correction. The actual display play time is called T2.actual (438).

[0194] In step 15 (440), the XR Application 404 queries for the pose associated with the updated prediction of the display time (T2. p red icted 2) . This can be achieved by calling the xrLocate Views function in OpenXR.

[0195] In step 16 (442), the XR Runtime 402 performs pose estimation.

[0196] In step 17 (444), the XR Runtime 402 returns a pose estimate (P.predicted2).

[0197] In step 18 (446), the XR Application 404 computes a pose error estimate (P.predictedl - P.predicted2) and a time error estimate (T2.predicted1 — T2.predicted2).Table 2. Rendering Depending on Confidence Level

[0198] In the example of Table 2, the confidence level threshold is 5 / 8, but other thresholds may be used in other embodiments.

[0199] In some embodiments, information regarding the confidence level of the predicted pose is inferred directly from the values of the flags, e.g. without calculating a specific numerical confidence level. For example, the confidence level of the predicted pose may be determined to be sufficiently high (e.g. at or above a threshold) in response to a determination that both the XR_VIEW_STATE_POSITION_VALID_BIT and the XR_VIEW_STATE_ORIENTATION_VALID_BIT are equal to “1”, whereas the confidence level of the predicted pose may be determined to be low (e.g. below the threshold) if either or both of those bits are equal to “0.” Alternatively, other logical combinations of these bits may be used to determine whether the confidence level of the predicted pose is sufficiently high to use that predicted pose for the rendering of a scene.

[0200] Other example embodiments may shift various processing features to the Edge Application Server, as shown in the following examples.

[0201] FIG. 5 is a message sequencing diagram illustrating an example process using pose confidence information with the confidence level checked by the edge application server according to some embodiments. FIG. 5 shows an example message sequence 500.

[0202] In step 1 (508), the XR Application 504 estimates the round-trip time (RTT) between the XR application 504 and the Edge Application Server (EAS) 506. In step 2 (510), the XR Application 504 queries for the next display time. This (and step 3) can be achieved by calling the xrWaitFrame function in embodiments that use OpenXR. In step 3 (512), the XR Runtime 502 replies with information indicating the next display time. In step 4 (514), the XR application 504 predicts a display time - an initial prediction - and the use of initial is because a second prediction / estimation will be made later. This predicted display time is called T2. predicted 1 . In step 5 (516), the XR application 504 queries for a predicted pose at the initial predicted display time T2.predicted1. Calling the function xrLocateViews in OpenXR can achieve this step and step 7. In step 6 (518), the XR Runtime 502 predicts the pose, and the prediction occurs at time T1. In step 7 (520), the XR Runtime 502 returns with the predicted pose (P.predictedl).

[0203] In step 7bis (522), the XR Application 504 computes the confidence level. In step 8 (524), that confidence level is sent to the Edge Application Server 506 together with the predicted pose (P.predictedl) and the predicted initial display time (T2.predicted1). In step 8bis (526), the Edge Application Server 506 checks the received confidence level. If confidence level is lower than a preconfigured threshold, for example 5 / 8, then, depending on the strategy, the Edge Application Server 506may take the previous rendered frame. Hence, the step 9 (528) is skipped, and step 10 (530) is executed with the previous rendering frame. Steps 11 (532), 12 (534), and 13 (536) are kept.

[0204] In step 11 (532), the XR Application 504 sends the rendered frame to the XR Runtime, e.g., via swapchain. This is achieved in some embodiments by calling the xrReleaseSwapchainlmage function in OpenXR. The XR Application passes the display time used for the rendering the frame, and this can be achieved by calling the xrEndFrame function in OpenXR. In step 12 (534), the XR Application 504 queries for the predicted display time. This is intended to get a more accurate prediction of the display time than the one in step 4, because there is less time to predict into the future at this moment. In step 13 (536), the XR Runtime 502 returns an updated prediction of the display time (T2.predicted2). In step 14 (538), the XR Runtime 502 performs reprojection for pose correction. The actual display play time is called T2.actual (540). In step 15 (542), the XR Application 504 queries for the pose associated with the updated prediction of the display time (T2.predicted2). This can be achieved by calling the xrLocateViews function in OpenXR. In step 16 (544), the XR Runtime 502 performs pose estimation. In step 17 (546), the XR Runtime 502 returns a pose estimate (P.predicted2). In step 18 (548), the XR Application 504 computes a pose error estimate (P.predictedl - P.predicted2) and a time error estimate (T2.predicted1 — T2.predicted2).

[0205] FIG. 6 is a message sequencing diagram illustrating an example process using pose confidence information with the confidence level computed and checked by the edge application server according to some embodiments. FIG. 6 shows an example message sequence 600.

[0206] In step 1 (608), the XR Application 604 estimates the round-trip time (RTT) between the XR application 404 and the Edge Application Server (EAS) 606. In step 2 (610), the XR Application 604 queries for the next display time. This (and step 3) can be achieved by calling the xrWaitFrame function in embodiments that use OpenXR. In step 3 (612), the XR Runtime 602 replies with information indicating the next display time. In step 4 (614), the XR application 604 predicts a display time - an initial prediction - and the use of initial is because a second prediction / estimation will be made later. This predicted display time is called T2. predicted 1 . In step 5 (616), the XR application 604 queries for a predicted pose at the initial predicted display time T2.predicted1. Calling the function xrLocateViews in OpenXR can achieve this step and step 7. In step 6 (618), the XR Runtime 602 predicts the pose, and the prediction occurs at time T1. In step 7 (620), the XR Runtime 602 returns with the predicted pose (P.predictedl).

[0207] In step 8 (622), the XR application 604 sends the predicted pose (P.predictedl) with the view’s status flags, e.g., XrViewStateFlags flags, and the predicted initial display time (T2.predicted1). In step 8bis, the Edge Application Server 606 computes the confidence level using the received view’s statusflags. The Edge Application Server 606 checks the confidence level. If confidence level is lower than a pre-configured threshold, for example 5 / 8. Then, depending on the strategy, the Edge Application Server 606 may take the previous rendered frame. Hence, step 9 (624) is skipped. In that case, step 10 (626) is executed with the previous rendering frame. The Edge Application Server 606 may send the computed Confidence level with the rendered frame which may be used to compute a consolidated confidence level on step 18 (644). Steps 11 (628), 12 (630), and 13 (632) are kept.

[0208] In step 11 (628), the XR Application 604 sends the rendered frame to the XR Runtime, e.g., via swapchain. This is achieved in some embodiments by calling the xrReleaseSwapchainlmage function in OpenXR. The XR Application passes the display time used for the rendering the frame, and this can be achieved by calling the xrEndFrame function in OpenXR. In step 12 (630), the XR Application 604 queries for the predicted display time. This is intended to get a more accurate prediction of the display time than the one in step 4, because there is less time to predict into the future at this moment. In step 13 (632), the XR Runtime 602 returns an updated prediction of the display time (T2.predicted2). In step 14 (634), the XR Runtime 602 performs reprojection for pose correction. The actual display play time is called T2.actual (636). In step 15 (638), the XR Application 404 queries for the pose associated with the updated prediction of the display time (T2.predicted2). This can be achieved by calling the xrLocateViews function in OpenXR. In step 16 (640), the XR Runtime 602 performs pose estimation. In step 17 (642), the XR Runtime 602 returns a pose estimate (P.predicted2). In step 18 (844), the XR Application 604 computes a pose error estimate (P.predictedl - P.predicted2) and a time error estimate (T2.predicted1 — T2.predicted2).

[0209] Stated differently, the differences between FIGs. 4, 5, and 6 are seen in steps 7bis, 8, 8bis, and 10 (in which step 7bis only applies to FIGs. 4 and 5 and step 8bis only applies to FIGs. 5 and 6 for some embodiments). Steps 1 to 7 are identical in FIGs. 4, 5, and 6. Also, step 9 is identical in FIGs. 4, 5, and 6. Furthermore, steps 11 to 18 are identical in FIGs. 4, 5, and 6. In some embodiments, the previously rendered frame may be used for display, and steps 8, 9, and 10 of FIGs. 4, 5, and 6 and, step 8bis of FIG. 6 if applicable, may be skipped, and step 11 of FIGs. 4, 5, and 6 is executed using the previously rendered frame.

[0210] For some embodiments, as shown in step 7bis of FIG. 4, the XR application computes and checks the confidence level. For some embodiments, as shown in step 7bis of FIG. 5, the XR application computes the confidence level. For some embodiments, as shown in FIG. 6, there is no step 7bis.

[0211] For some embodiments, as shown in step 8 of FIG. 4, the XR application sends to the edge application server the predicted pose P.predictedl and the predicted initial display time (T2.predicted1).For some embodiments, as shown in step 8 of FIG. 5, the XR application sends to the edge application server the predicted pose P.predictedl with the confidence level and the predicted initial display time (T2.predicted1). For some embodiments, as shown in step 8 of FIG. 6, the XR application sends to the edge application server the predicted pose P.predictedl with the view’s status XrViewStateFlags flags and the predicted initial display time (T2.predicted1).

[0212] For some embodiments, as shown in FIG. 4, there is no step 8bis. For some embodiments, as shown in step 8bis of FIG. 5, the edge application server checks the confidence level. For some embodiments, as shown in step 8bis of FIG. 6, the edge application server computes and checks the confidence level.

[0213] For some embodiments, as shown in step 10 of FIG. 4, the edge application server returns to the XR application the rendered frame and the predicted initial display time (T2. predicted 1 ). For some embodiments, as shown in step 10 of FIG. 5, the edge application server returns to the XR application the rendered frame and the predicted initial display time (T2.predicted1). For some embodiments, as shown in step 10 of FIG. 6, the edge application server returns to the XR application the rendered frame, the predicted initial display time (T2.predicted1), and the confidence level.

[0214] FIG. 7 is a flowchart illustrating an example process including frame selection using pose confidence information according to some embodiments. FIG. 7 illustrates a method performed in some embodiments. At 702, a first predicted display time is obtained, the first predicted display time representing a first prediction of a time at which a particular frame will be displayed. The prediction may be based on the frame rate and on an estimate of the round-trip time (RTT) for obtaining a rendered frame.

[0215] At 704, a first predicted pose is obtained. The first predicted pose represents a prediction of what the user pose will be at the first predicted display time. First pose confidence information is also obtained, the first pose confidence information representing a confidence level for the first predicted pose. The first predicted pose and the first pose confidence information may be obtained in some embodiments by calling the function xrLocateViewsQ in an OpenXR system, or by other techniques using other XR runtime systems.

[0216] At 706, a determination is made of whether the first pose confidence information indicates a sufficiently high confidence level. In some embodiments, this may be performed by converting values of the flags XrViewStateFlags into a numerical confidence level and determining whether the numerical confidence level is at or above a threshold. In some embodiments, this may be performed by performing a logical operation on the values of some or all of the XrViewStateFlags. For example, the poseconfidence may be determined to be high if and only if the XR_VIEW_STATE_POSITION_VALID_BIT and the XR_VIEW_STATE_ORIENTATION_VALID_BIT are both equal to “1”.

[0217] If the pose confidence level is sufficiently high, then at 708, a new rendered frame is obtained based on the first predicted pose. This may be done in some embodiments by sending the first predicted pose and the first predicted display time to an edge application server and receiving the result from the server. At 710, a second predicted display time is obtained. The second predicted display time may be obtained by calling xrWaitFrameQ in an OpenXR system, or by other techniques using other XR runtime systems. At 712, the new rendered frame is reprojected based on the second predicted display time. At 718, the reprojected frame is caused to be displayed to the user, for example by providing the reprojected frame to user equipment for display and / or by actually displaying the reprojected frame (if the system performing the method of FIG. 7 has built-in display capabilities).

[0218] If, on the other hand, the pose confidence level is not sufficiently high, then a newly-rendered frame is not obtained based on the first predicted pose. At 714, a second predicted display time is obtained. The second predicted display time may be obtained by calling xrWaitFrameQ in an OpenXR system, or by other techniques using other XR runtime systems. At 716, a previously rendered frame is reprojected based on the second predicted display time. At 718, the reprojected frame is caused to be displayed to the user.

[0219] FIG. 8 is a flowchart illustrating an example process including frame selection using pose confidence information according to some embodiments. An example embodiment may further be described with respect to FIG. 8. At 802, a first predicted display time is obtained, for example as in steps 4 (214, 414, 514, 614) and 702. At 804, a first predicted pose is obtained. The first predicted pose represents a prediction of what the user pose will be at the first predicted display time. First pose confidence information is also obtained, the first pose confidence information representing a confidence level for the first predicted pose. At 806, a frame to be displayed is selected based on the first pose confidence level information. For example, in response to a determination that the first pose confidence level information is below a threshold, a previously rendered frame may be selected. In response to a determination that the first pose confidence level information is greater than or equal to a threshold, a new frame may be rendered (e.g. by an EAS) based on the first predicted pose information. At 808, a second predicted display time is obtained. The second predicted display time may be obtained by calling xrWaitFrameQ in an OpenXR system, or by other techniques using other XR runtime systems. At 810, the selected frame (which may be a newly rendered frame or a previously-rendered frame depending on the selection) is reprojected based on the second predicted display time. At 812, the reprojected frame is caused to be displayed to the user.Use of Confidence Information in Pose Error Measurement

[0220] FIG. 9 is a flowchart illustrating an example process including a pose error determination using pose confidence information according to some embodiments. In some embodiments, a process for pose error measurement proceeds based on the pose confidence levels as obtained above. An example process is described with respect to FIG. 9.

[0221] The method illustrated in FIG. 9 proceeds analogously to the method shown in FIG. 8. At 902, a first predicted display time is obtained. At 904, a first predicted pose is obtained. At 906, a frame to be displayed is selected based on the first pose confidence level information. At 908, a second predicted display time is obtained. At 910, the selected frame (which may be a newly rendered frame or a previously-rendered frame depending on the selection) is reprojected based on the second predicted display time. At 912, the reprojected frame is caused to be displayed to the user.

[0222] In addition, at 914, a second predicted pose is obtained. The second predicted pose represents a prediction of the user pose at the second predicted display time. Second pose confidence information is also obtained, the second pose confidence information representing a confidence level for the second predicted pose. The second predicted pose and the second pose confidence information may be obtained in some embodiments by calling the function xrLocateViewsQ in an OpenXR system, or by other techniques using other XR runtime systems.

[0223] At 916, a pose error may be determined. The pose error may be determined by calculating a difference between the first predicted pose and the second predicted pose, for example using (P.predictedl - P.predicted2).

[0224] In some embodiments, at step 18 (446, 548, 644), a determination is made of whether or not to calculate to pose error based on the first pose confidence information (e.g., as returned at step 7 (420, 520, 620)) and / or the second pose confidence information (e.g., as returned at step 17 (444, 546, 642)). In some embodiments, at step 18 (446, 548, 644), the pose error is calculated, and a determination is made based on the first pose confidence information and / or the second pose confidence information of whether or not the pose error is determined to be valid.

[0225] In some embodiments, a bitwise product is calculated of the first pose confidence information and the second pose confidence information. For example, final flag values may be determined as follows:POSITION _VALID_BITfinal=POSITION _VALID_BITstep7& POSITION _VALID_BITstepl7ORIENTATION _VALID_BITfinal=ORIENT ATON_VALID_BITstep7& ORIENTATION VALID J ITstepl7POSITION _TRACKED_BITfinal=POSITION _TRACKED_BITstep7& POSITION _TRACKED_BITstepl7ORIENTATION_TRACKED_BITfinal=ORIENT AT ON _TRACKED_BITstep7& ORIENTATION TRACKED JITstepl7

[0226] In some embodiments, a confidence level is determined from the resulting final flag values using the confidence levels as indicated in Table 1, though other confidence values may be assigned in other embodiments. In some embodiments, the validity (or invalidity) of the pose error measurement is determined based on the numerical confidence level as shown in Table 3. In some embodiments, a determination of whether or not to calculate the pose error measurement is made based on the numerical confidence level as shown in Table 3.Table 3. Pose Error Measurement Depending on Confidence Level

[0227] Thus, if the confidence levels from step 7 and step 17 are both equal to 1 , the measure is valid. In some embodiments, a low level of confidence is associated with the pose error measurement if the product of confidence level from step 7 and step 17 is greater than or equal to a threshold (e.g. 5 / 8 in Table 3). In some embodiments, the pose error measurement is considered not to be valid if the result is lower than this threshold.

[0228] In some embodiments, one or more of steps 15, 16, 17, and 18 are omitted if the first pose confidence information (e.g. obtained in step 7) indicates a confidence level that is lower than a threshold (e.g. lower than 5 / 8). In some embodiments, the pose measurement error is computed (e.g. at step 18) if the confidence level is higher than or equal to a threshold (such as 5 / 8).

[0229] The level of confidence and the result of the computation of the pose error may be provided to the application. Depending on this, application can decide to adjust (or not) the process.

[0230] In some embodiments, time error estimation is unchanged and proceeds as shown in FIG. 2.

[0231] FIG. 10 is a flowchart illustrating an example process for rendering a scene based on a confidence level of a predicted pose according to some embodiments. For some embodiments, an example process 1000 may include obtaining 1002 a first predicted frame display time. For some embodiments, the example process 1000 may further include obtaining 1004 first predicted pose information representing a prediction of a user pose at the first predicted frame display time. For some embodiments, the example process 1000 may further include determining 1006 first pose confidence information indicating a confidence level of the first predicted pose information. For some embodiments, the example process 1000 may further include selecting 1008 a frame for display based at least in part on the first pose confidence information. For some embodiments, the example process 1000 may further include causing 1010 display of the selected frame.

[0232] FIG. 11 is a flowchart illustrating an example process for rendering a scene based on a confidence level of a predicted pose according to some embodiments. For some embodiments, an example process 1100 may include obtaining 1102 a first predicted frame display time. For some embodiments, the example process 1100 may further include obtaining 1104 first predicted pose information representing a prediction of a user pose at the first predicted frame display time. For some embodiments, the example process 1100 may further include determining 1106 first pose confidence information indicating a confidence level of the first predicted pose information. For some embodiments, the example process 1100 may further include communicating 1108, to an edge application server, the determined first pose confidence information indicating the confidence level of the first predicted pose information. For some embodiments, the example process 1100 may further include selecting 1110 aframe for display based at least in part on the first pose confidence information. For some embodiments, the example process 1100 may further include causing 1112 display of the selected frame.

[0233] FIG. 12 is a flowchart illustrating an example process for rendering a scene based on a confidence level of a predicted pose according to some embodiments. For some embodiments, an example process 1200 may include receiving 1202 a first predicted frame display time. For some embodiments, the example process 1200 may further include receiving 1204 first predicted pose information representing a prediction of a user pose at the first predicted frame display time. For some embodiments, the example process 1200 may further include receiving 1206 first pose confidence information indicating a confidence level of the first predicted pose information. For some embodiments, the example process 1200 may further include selecting 1208 a frame for display based at least in part on the first pose confidence information. For some embodiments, the example process may further include rendering 1210 the selected frame. For some embodiments, the example process 1200 may further include sending 1212, to an extended reality (XR) application device, the rendered frame. For some embodiments, the example process 1200 may further include sending, to the XR application device, the first predicted frame display time. For some embodiments, the XR application device already knows the first predicted frame display time and keeps track of this value for later use.

[0234] FIG. 13 is a flowchart illustrating an example process for rendering a scene based on a confidence level of a predicted pose according to some embodiments. For some embodiments, an example process 1300 may include obtaining 1302 a first predicted frame display time. For some embodiments, the example process 1300 may further include obtaining 1304 first predicted pose information representing a prediction of a user pose at the first predicted frame display time. For some embodiments, the example process 1300 may further include communicating 1306, to an edge application server, the first predicted frame display time, the first predicted pose information, and information indicating an extended reality (XR) view state. For some embodiments, the example process 1300 may further include obtaining 1308 first pose confidence information indicating a confidence level of the first predicted pose information. For some embodiments, the example process 1300 may further include selecting 1310 a frame for display based at least in part on the first pose confidence information. For some embodiments, the example process 1300 may further include causing 1312 display of the selected frame.

[0235] FIG. 14 is a flowchart illustrating an example process for rendering a scene based on a confidence level of a predicted pose according to some embodiments. For some embodiments, an example process 1400 may include receiving 1402 a first predicted frame display time. For some embodiments, the example process 1400 may further include receiving 1404 first predicted poseinformation representing a prediction of a user pose at the first predicted frame display time. For some embodiments, the example process 1400 may further include receiving 1406 information indicating an extended reality (XR) view state. For some embodiments, the example process 1400 may further include determining 1408 first pose confidence information indicating a confidence level of the first predicted pose information. For some embodiments, the example process 1400 may further include selecting 1410 a frame for display based at least in part on the first pose confidence information. For some embodiments, the example process 1400 may further include rendering 1412 the selected frame. For some embodiments, the example process 1400 may further include sending 1414, to an extended reality (XR) application device, the rendered frame, and the determined first pose confidence information. For some embodiments, the example process 1400 may further include sending, to the XR application device, the first predicted frame display time. For some embodiments, the XR application device already knows the first predicted frame display time and keeps track of this value for later use.

[0236] While the methods and systems in accordance with some embodiments are generally discussed in context of extended reality (XR), some embodiments may be applied to any XR contexts such as, e.g., virtual reality (VR) / mixed reality (MR) / augmented reality (AR) contexts. Also, although the term “head mounted display (HMD)” is used herein in accordance with some embodiments, some embodiments may be applied to a wearable device (which may or may not be attached to the head) capable of, e.g., XR, VR, AR, and / or MR for some embodiments.

[0237] A first example method in accordance with some embodiments may include: obtaining a first predicted frame display time; obtaining first predicted pose information representing a prediction of a user pose at the first predicted frame display time; obtaining first pose confidence information indicating a confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; and causing display of the selected frame.

[0238] Some embodiments of the first example method may further include: obtaining a second predicted frame display time; and performing reprojection of the selected frame based on the second predicted frame display time before causing display of the selected frame.

[0239] For some embodiments of the first example method, in response to a determination that the first pose confidence information indicates a confidence level at least as great as a threshold, a newly- rendered frame is selected as the selected frame to be displayed.

[0240] Some embodiments of the first example method may further include rendering the selected frame based on the first predicted pose.

[0241] For some embodiments of the first example method, in response to a determination that the first pose confidence information indicates a confidence level below a threshold, a previously-rendered frame is selected as the selected frame to be displayed.

[0242] For some embodiments of the first example method, the first pose confidence information comprises a plurality of flags.

[0243] For some embodiments of the first example method, the first pose confidence information comprises at least a position validity flag and an orientation validity flag.

[0244] For some embodiments of the first example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame may be selected as the selected frame to be displayed.

[0245] For some embodiments of the first example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously- rendered frame may be selected as the selected frame to be displayed.

[0246] For some embodiments of the first example method, the first pose confidence information may include at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

[0247] Some embodiments of the first example method may further include: obtaining second predicted pose information representing a prediction of a user pose at a display time of the selected frame; and obtaining second pose confidence information indicating a confidence level of the second predicted pose information.

[0248] Some embodiments of the first example method may further include calculating a pose error based on a difference between the first predicted pose information and the second predicted pose information.

[0249] Some embodiments of the first example method may further include determining, based on the first pose confidence information and the second pose confidence information, whether to calculate the pose error, the pose error being calculated in response to a determination to calculate the pose error.

[0250] Some embodiments of the first example method may further include determining, based on the first pose confidence information and the second pose confidence information, whether the pose error is valid.

[0251] For some embodiments of the first example method, obtaining the first pose confidence information comprises determining the first pose confidence information.

[0252] A first example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0253] A second example method in accordance with some embodiments may include: obtaining a first predicted frame display time; obtaining first predicted pose information representing a prediction of a user pose at the first predicted frame display time; determining first pose confidence information indicating a confidence level of the first predicted pose information; communicating, to an edge application server, the determined first pose confidence information indicating the confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; and causing display of the selected frame.

[0254] Some embodiments of the second example method may further include: obtaining a second predicted frame display time; and performing reprojection of the selected frame based on the second predicted frame display time before causing display of the selected frame.

[0255] For some embodiments of the second example method, the first pose confidence information may include a plurality of flags.

[0256] For some embodiments of the second example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag.

[0257] For some embodiments of the second example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

[0258] For some embodiments of the second example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously-rendered frame is selected as the selected frame to be displayed.

[0259] For some embodiments of the second example method, the first pose confidence information may include at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

[0260] Some embodiments of the second example method may further include: obtaining second predicted pose information representing a prediction of a user pose at a display time of the selected frame; and obtaining second pose confidence information indicating a confidence level of the second predicted pose information.

[0261] Some embodiments of the second example method may further include calculating a pose error based on a difference between the first predicted pose information and the second predicted pose information.

[0262] Some embodiments of the second example method may further include determining, based on the first pose confidence information and the second pose confidence information, whether to calculate the pose error, the pose error being calculated in response to a determination to calculate the pose error.

[0263] Some embodiments of the second example method may further include determining, based on the first pose confidence information and the second pose confidence information, whether the pose error is valid.

[0264] A second example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0265] A third example method in accordance with some embodiments may include: receiving a first predicted frame display time; receiving first predicted pose information representing a prediction of a user pose at the first predicted frame display time; receiving first pose confidence information indicating a confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; rendering the selected frame; and sending, to an extended reality (XR) application device, the rendered frame.

[0266] For some embodiments of the third example method, in response to a determination that the first pose confidence information indicates a confidence level at least as great as a threshold, a newly- rendered frame is selected as the selected frame to be displayed.

[0267] For some embodiments of the third example method, rendering the selected frame is based on the first predicted pose.

[0268] For some embodiments of the third example method, in response to a determination that the first pose confidence information indicates a confidence level below a threshold, a previously-rendered frame is selected as the selected frame to be displayed.

[0269] For some embodiments of the third example method, the first pose confidence information may include a plurality of flags.

[0270] For some embodiments of the third example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag.

[0271] For some embodiments of the third example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

[0272] For some embodiments of the third example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously-rendered frame is selected as the selected frame to be displayed.

[0273] For some embodiments of the third example method, the first pose confidence information may include at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

[0274] Some embodiments of the third example method may further include sending, to the extended reality (XR) application device, the first predicted frame display time.

[0275] A third example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0276] A fourth example method in accordance with some embodiments may include: obtaining a first predicted frame display time; obtaining first predicted pose information representing a prediction of a user pose at the first predicted frame display time; communicating, to an edge application server, the first predicted frame display time, the first predicted pose information, and information indicating an extended reality (XR) view state; obtaining first pose confidence information indicating a confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; and causing display of the selected frame.

[0277] Some embodiments of the fourth example method may further include: obtaining a second predicted frame display time; and performing reprojection of the selected frame based on the second predicted frame display time before causing display of the selected frame.

[0278] For some embodiments of the fourth example method, the first pose confidence information may include a plurality of flags.

[0279] For some embodiments of the fourth example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag.

[0280] For some embodiments of the fourth example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

[0281] For some embodiments of the fourth example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously-rendered frame is selected as the selected frame to be displayed.

[0282] For some embodiments of the fourth example method, the first pose confidence information may include at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

[0283] Some embodiments of the fourth example method may further include: obtaining second predicted pose information representing a prediction of a user pose at a display time of the selected frame; and obtaining second pose confidence information indicating a confidence level of the second predicted pose information.

[0284] Some embodiments of the fourth example method may further include calculating a pose error based on a difference between the first predicted pose information and the second predicted pose information.

[0285] Some embodiments of the fourth example method may further include determining, based on the first pose confidence information and the second pose confidence information, whether to calculate the pose error, the pose error being calculated in response to a determination to calculate the pose error.

[0286] Some embodiments of the fourth example method may further include determining, based on the first pose confidence information and the second pose confidence information, whether the pose error is valid.

[0287] A fourth example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0288] A fifth example method in accordance with some embodiments may include: receiving a first predicted frame display time; receiving first predicted pose information representing a prediction of a user pose at the first predicted frame display time; receiving information indicating an extended reality (XR) view state; determining first pose confidence information indicating a confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; rendering the selected frame; and sending, to an extended reality (XR) application device, the rendered frame, and the determined first pose confidence information.

[0289] For some embodiments of the firth example method, in response to a determination that the first pose confidence information indicates a confidence level at least as great as a threshold, a newly- rendered frame is selected as the selected frame to be displayed.

[0290] For some embodiments of the firth example method, rendering the selected frame is based on the first predicted pose.

[0291] For some embodiments of the firth example method, in response to a determination that the first pose confidence information indicates a confidence level below a threshold, a previously-rendered frame is selected as the selected frame to be displayed.

[0292] For some embodiments of the firth example method, the first pose confidence information may include a plurality of flags.

[0293] For some embodiments of the firth example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag.

[0294] For some embodiments of the firth example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

[0295] For some embodiments of the firth example method, the first pose confidence information may include at least a position validity flag and an orientation validity flag, and in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously- rendered frame is selected as the selected frame to be displayed.

[0296] For some embodiments of the firth example method, the first pose confidence information may include at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

[0297] Some embodiments of the fifth example method may further include sending, to the extended reality (XR) application device, the first predicted frame display time.

[0298] A fifth example apparatus in accordance with some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform any one of the methods listed above.

[0299] A sixth example method / apparatus in accordance with some embodiments may include one or more processors configured to perform any one of the methods listed above.

[0300] A seventh example method / apparatus in accordance with some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any one of the methods listed above.

[0301] An eighth example method / apparatus in accordance with some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any one of the methods listed above.

[0302] An example computer-readable medium in accordance with some embodiments may include instructions for causing one or more processors to perform any one of the methods listed above.

[0303] For some embodiments of the example computer-readable medium, the computer-readable medium is a non-transitory storage medium.

[0304] An example computer program in accordance with some embodiments may include instructions which, when the program is executed by one or more processors, causes the one or more processors to carry out any one of the methods listed above.

[0305] An example signal in accordance with some embodiments may include a scene description file generated according to any one of the methods listed above.

[0306] This disclosure describes a variety of aspects, including tools, features, embodiments, models, approaches, etc. Many of these aspects are described with specificity and, at least to show the individual characteristics, are often described in a manner that may sound limiting. However, this is for purposes of clarity in description, and does not limit the disclosure or scope of those aspects. Indeed, all of the different aspects can be combined and interchanged to provide further aspects. Moreover, the aspects can be combined and interchanged with aspects described in earlier filings as well.

[0307] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are illustrated specifically, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of theimplementations. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects can be implemented as a method, an apparatus, a computer readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods described, and / or a computer readable storage medium having stored thereon a bitstream generated according to any of the methods described.

[0308] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., such as, for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0309] Various numeric values may be used in the present disclosure, for example. The specific values are for example purposes and the aspects described are not limited to these specific values.

[0310] Embodiments described herein may be carried out by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The processor can be of any type appropriate to the technical environment and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0311] When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of a corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of a corresponding method / process.

[0312] The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, a processor, which refers toprocessing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0313] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0314] Additionally, this disclosure may refer to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0315] Further, this disclosure may refer to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0316] Additionally, this disclosure may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0317] It is to be appreciated that the use of any of the following 7”, “and / or”, and “at least one of’, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the firstand third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended for as many items as are listed.

[0318] Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments the encoder signals a particular one of a plurality of parameters for region-based filter parameter selection for de-artifact filtering. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various embodiments. It is to be appreciated that signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0319] Implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0320] We describe a number of embodiments. Features of these embodiments can be provided alone or in any combination, across various claim categories and types. Further, embodiments can include one or more of the following features, devices, or aspects, alone or in any combination, across various claim categories and types:• A bitstream or signal that includes one or more of the described syntax elements, or variations thereof.• A bitstream or signal that includes syntax conveying information generated according to any of the embodiments described.• Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements, or variations thereof.• Creating and / or transmitting and / or receiving and / or decoding according to any of the embodiments described.• A method, process, apparatus, medium storing instructions, medium storing data, or signal according to any of the embodiments described.

[0321] Note that various hardware elements of one or more of the described embodiments are referred to as “modules” that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions could take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.

[0322] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

CLAIMS1. A method comprising: obtaining a first predicted frame display time; obtaining first predicted pose information representing a prediction of a user pose at the first predicted frame display time; obtaining first pose confidence information indicating a confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; and causing display of the selected frame.

2. The method of claim 1 , further comprising: obtaining a second predicted frame display time; and performing reprojection of the selected frame based on the second predicted frame display time before causing display of the selected frame.

3. The method of any one of claims 1-2, wherein: in response to a determination that the first pose confidence information indicates a confidence level at least as great as a threshold, a newly-rendered frame is selected as the selected frame to be displayed.

4. The method of claim 3, further comprising rendering the selected frame based on the first predicted pose.

5. The method of any one of claims 1 -2, wherein: in response to a determination that the first pose confidence information indicates a confidence level below a threshold, a previously-rendered frame is selected as the selected frame to be displayed.

6. The method of any one of claims 1-5, wherein the first pose confidence information comprises a plurality of flags.

7. The method of any one of claims 1-6, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag.

8. The method of any one of claims 1 -7, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag, and wherein in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

9. The method of any one of claims 1 -8, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag, and wherein in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously-rendered frame is selected as the selected frame to be displayed.

10. The method of any one of claims 1-9, wherein the first pose confidence information comprises at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.11 . The method of any one of claims 1-10, further comprising: obtaining second predicted pose information representing a prediction of a user pose at a display time of the selected frame; and obtaining second pose confidence information indicating a confidence level of the second predicted pose information.

12. The method of claim 11 , further comprising calculating a pose error based on a difference between the first predicted pose information and the second predicted pose information.

13. The method of claim 12, further comprising determining, based on the first pose confidence information and the second pose confidence information, whether to calculate the pose error, the pose error being calculated in response to a determination to calculate the pose error.

14. The method of claim 12, further comprising determining, based on the first pose confidence information and the second pose confidence information, whether the pose error is valid.

15. The method of any one of claims 1-14, wherein obtaining the first pose confidence information comprises determining the first pose confidence information.

16. An apparatus comprising: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform the method of any one of claims 1 through 15.

17. A method comprising: obtaining a first predicted frame display time; obtaining first predicted pose information representing a prediction of a user pose at the first predicted frame display time; determining first pose confidence information indicating a confidence level of the first predicted pose information; communicating, to an edge application server, the determined first pose confidence information indicating the confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; and causing display of the selected frame.

18. The method of claim 17, further comprising: obtaining a second predicted frame display time; and performing reprojection of the selected frame based on the second predicted frame display time before causing display of the selected frame.

19. The method of any one of claims 17-18, wherein the first pose confidence information comprises a plurality of flags.

20. The method of any one of claims 17-19, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag.21 . The method of any one of claims 17-20, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag, and wherein in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

22. The method of any one of claims 17-21 , wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag, and wherein in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously-rendered frame is selected as the selected frame to be displayed.

23. The method of any one of claims 17-22, wherein the first pose confidence information comprises at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

24. The method of any one of claims 17-23, further comprising: obtaining second predicted pose information representing a prediction of a user pose at a display time of the selected frame; and obtaining second pose confidence information indicating a confidence level of the second predicted pose information.

25. The method of claim 24, further comprising calculating a pose error based on a difference between the first predicted pose information and the second predicted pose information.

26. The method of claim 25, further comprising determining, based on the first pose confidence information and the second pose confidence information, whether to calculate the pose error, the pose error being calculated in response to a determination to calculate the pose error.

27. The method of claim 26, further comprising determining, based on the first pose confidence information and the second pose confidence information, whether the pose error is valid.

28. An apparatus comprising: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform the method of any one of claims 17 through 27.

29. A method comprising: receiving a first predicted frame display time; receiving first predicted pose information representing a prediction of a user pose at the first predicted frame display time; receiving first pose confidence information indicating a confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; rendering the selected frame; and sending, to an extended reality (XR) application device, the rendered frame.

30. The method of claim 29, wherein: in response to a determination that the first pose confidence information indicates a confidence level at least as great as a threshold, a newly-rendered frame is selected as the selected frame to be displayed.31 . The method of claim 30, wherein rendering the selected frame is based on the first predicted pose.

32. The method of any one of claims 30-31 , wherein: in response to a determination that the first pose confidence information indicates a confidence level below a threshold, a previously-rendered frame is selected as the selected frame to be displayed.

33. The method of any one of claims 30-32, wherein the first pose confidence information comprises a plurality of flags.

34. The method of any one of claims 30-33, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag.

35. The method of any one of claims 30-34, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag, and wherein in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

36. The method of any one of claims 30-35, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag, and wherein in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously-rendered frame is selected as the selected frame to be displayed.

37. The method of any one of claims 30-36, wherein the first pose confidence information comprises at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

38. The method of any one of claims 30-37, further comprising sending, to the extended reality (XR) application device, the first predicted frame display time.

39. An apparatus comprising: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform the method of any one of claims 30 through 38.

40. A method comprising: obtaining a first predicted frame display time; obtaining first predicted pose information representing a prediction of a user pose at the first predicted frame display time; communicating, to an edge application server, the first predicted frame display time, the first predicted pose information, and information indicating an extended reality (XR) view state; obtaining first pose confidence information indicating a confidence level of the first predicted pose information;selecting a frame for display based at least in part on the first pose confidence information; and causing display of the selected frame.41 . The method of claim 40, further comprising: obtaining a second predicted frame display time; and performing reprojection of the selected frame based on the second predicted frame display time before causing display of the selected frame.

42. The method of any one of claims 40-41 , wherein the first pose confidence information comprises a plurality of flags.

43. The method of any one of claims 40-42, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag.

44. The method of any one of claims 40-43, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag, and wherein in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

45. The method of any one of claims 40-44, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag, and wherein in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously-rendered frame is selected as the selected frame to be displayed.

46. The method of any one of claims 40-45, wherein the first pose confidence information comprises at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

47. The method of any one of claims 40-46, further comprising:obtaining second predicted pose information representing a prediction of a user pose at a display time of the selected frame; and obtaining second pose confidence information indicating a confidence level of the second predicted pose information.

48. The method of claim 47, further comprising calculating a pose error based on a difference between the first predicted pose information and the second predicted pose information.

49. The method of claim 48, further comprising determining, based on the first pose confidence information and the second pose confidence information, whether to calculate the pose error, the pose error being calculated in response to a determination to calculate the pose error.

50. The method of claim 48, further comprising determining, based on the first pose confidence information and the second pose confidence information, whether the pose error is valid.51 . An apparatus comprising: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform the method of any one of claims 40 through 50.

52. A method comprising: receiving a first predicted frame display time; receiving first predicted pose information representing a prediction of a user pose at the first predicted frame display time; receiving information indicating an extended reality (XR) view state; determining first pose confidence information indicating a confidence level of the first predicted pose information; selecting a frame for display based at least in part on the first pose confidence information; rendering the selected frame; and sending, to an extended reality (XR) application device, the rendered frame, and the determined first pose confidence information.

53. The method of claim 52, wherein:in response to a determination that the first pose confidence information indicates a confidence level at least as great as a threshold, a newly-rendered frame is selected as the selected frame to be displayed.

54. The method of claim 53, wherein rendering the selected frame is based on the first predicted pose.

55. The method of claim 52, wherein: in response to a determination that the first pose confidence information indicates a confidence level below a threshold, a previously-rendered frame is selected as the selected frame to be displayed.

56. The method of any one of claims 52-55, wherein the first pose confidence information comprises a plurality of flags.

57. The method of any one of claims 52-56, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag.

58. The method of any one of claims 52-57, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag, and wherein in response to a determination that the position validity flag and the orientation validity flag are both set, a newly-rendered frame is selected as the selected frame to be displayed.

59. The method of any one of claims 52-58, wherein the first pose confidence information comprises at least a position validity flag and an orientation validity flag, and wherein in response to a determination that at least one of the position validity flag and the orientation validity flag is not set, a previously-rendered frame is selected as the selected frame to be displayed.

60. The method of any one of claims 52-59, wherein the first pose confidence information comprises at least a position validity flag, an orientation validity flag, a position tracking flag, and an orientation tracking flag.

61. The method of any one of claims 52-60, further comprising sending, to the extended reality (XR) application device, the first predicted frame display time.

62. An apparatus comprising: a processor; and a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to perform the method of any one of claims 52 through 60.

63. An apparatus comprising one or more processors configured to perform the method of any of claims1-15, 17-27, 29-38, 40-50, and 52-61.

64. An apparatus comprising a computer-readable medium storing instructions for causing one or more processors to perform the method of any one of claims1-15, 17-27, 29-38, 40-50, and 52-61.

65. An apparatus comprising at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform the method of any one of claims1-15, 17-27, 29-38, 40-50, and 52-61.

66. A computer-readable medium including instructions for causing one or more processors to perform the method of any of claims1-15, 17-27, 29-38, 40-50, and 52-61.

67. The computer-readable medium of claim 66, wherein the computer-readable medium is a non- transitory storage medium.

68. A computer program product including instructions which, when the program is executed by one or more processors, causes the one or more processors to carry out the method of any of claims 1 -15, 17-27, 29-38, 40-50, and 52-61.

69. A signal including a scene description file generated according to any one of claims1-15, 17-27, 29- 38, 40-50, and 52-61.

Images