Method for Measuring the Experience Quality of XR Interactivity

Abstract

Some embodiments of the method may include: managing the processing of received interactions relating to an MR scene by configuring a metadata structure, the metadata structure comprising a set of time information identifications when processing received interactions; receiving user actions and initiating the processing of interaction requests based on user actions; identifying first time information in a first stage of processing interaction requests; updating interaction information with the first time information in a first stage; sending the interaction request to a server with the updated interaction information; and receiving a response to the interaction request, the response comprising the updated interaction information and further time information from the server according to the metadata structure; and identifying one or more quality of experience (QoE) metrics at least in part based on the updated further interaction information.

Description

Technical Field

[0001] Cross - reference to Related Applications

[0001] This application claims the benefit of European Patent Application Publication No. 23305520, filed on April 7, 2023, with the title "METHODS FOR MEASUREMENT OF XR INTERACTIVITY QUALITY OF EXPERIENCE", and incorporates all of it by reference herein.

Background Art

[0002] Background

[0002] Extended Reality (XR) is a technology that enables an interactive experience in which the real - world environment and / or video content are enhanced by virtual content that can be defined across multiple sensory modalities including vision, hearing, touch, etc. When an application is executed, virtual content (e.g., 3D content or audio / video files, etc.) is rendered in real - time in a way that matches the user context (environment, viewpoint, device, etc.). A scene graph (e.g., those proposed by Khronos / glTF, and its extensions defined in the MPEG Scene Description format or Apple / USDZ) is a possible way to represent the rendered content. They combine, on the one hand, a declarative description of the scene structure that links real - world environment objects and virtual objects, and on the other hand, a binary representation of the virtual content.

[0003]

[0003] Such a scene description framework ensures that time - stamped media and corresponding related virtual content are always available during the rendering of an application, but such a framework does not provide an explanation of how a user can interact with scene objects at runtime for an immersive XR experience. Therefore, user - specific XR experiences for engaging with immersive media are not supported.

Summary of the Invention

[0004] overview

[0004] One embodiment of the method according to several embodiments includes a client device communicating with a server to configure a metadata structure to manage the processing of received interactions relating to a mixed reality (MR) scene, the metadata structure comprising a series of time information identifications performed by the client device and the server device when processing received interactions; receiving a user action relating to an MR scene and initiating processing of an interaction request based on the user action according to the interaction information and metadata structure corresponding to the user action; the client device identifying first time information in a first stage of processing the interaction request; updating the interaction information with the first time information in a first stage; sending the interaction request to the server along with the updated interaction information; and receiving a response from the server to the interaction request, the response comprising further interaction information comprising the updated interaction information and further time information from the server according to the metadata structure; and identifying one or more quality of experience (QoE) metrics at least in part based on the updated further interaction information.

[0005]

[0005] Some embodiments of the method of the embodiment may further include identifying second time information in a second step of processing interaction requests in a client device, and updating interaction information with the second time information in the second step.

[0006]

[0006] In some embodiments of the method of the embodiment, at least one of the first time information and the second time information may include one or more timestamps.

[0007]

[0007] Some embodiments of the method of the embodiment may further include, in the client device, identifying second time information in a second stage of processing the response from the server, and updating further interaction information from the response from the server with the second time information in the second stage.

[0008]

[0008] Some embodiments of the method of the embodiment may further include receiving an identifier code and identifying the configuration of a metadata structure based on the received identifier code.

[0009]

[0009] Some embodiments of the method of the embodiment may further include receiving an identifier code and using the identifier code to check the processing order of user interactions on the server.

[0010]

[0010] Some embodiments of the method of the embodiment may further include receiving an identifier code and using the identifier code to identify a round-trip interaction delay.

[0011]

[0011] In some embodiments of the method of the embodiment, identifying the round-trip interaction delay may include storing the time the user performed the interaction associated with the identifier code.

[0012]

[0012] Some embodiments of the methods of the examples may further include specifying an interaction request drop rate, the interaction request drop rate may include the rate at which interaction requests are dropped by the server, and the interaction request drop rate may include a QoE metric.

[0013]

[0013] In some embodiments of the method of the example, at least one of the one or more QoE metrics may be round-trip interaction delay.

[0014]

[0014] For some embodiments of the method of the example, the response may include an updated description of the MR scene.

[0015]

[0015] In some embodiments of the method of the example, the response may include a new description of the MR scene.

[0016]

[0016] In some embodiments of the method of the embodiment, the response may include information indicating that the interaction request was dropped by the server.

[0017]

[0017] In some embodiments of the method of the embodiment, the response may include an encoded version of the rendered MR scene.

[0018]

[0018] In some embodiments of the method of the embodiment, further time information may be included from the processing stage on the server.

[0019]

[0019] An embodiment of the apparatus according to several embodiments may include a processor and a non-temporary computer-readable medium that stores instructions that, when executed by the processor, are operable to cause the apparatus to perform one of the methods listed above.

[0020]

[0020] A method of an additional embodiment according to several embodiments may include a server receiving communications from a client device and managing the processing of user interactions relating to a mixed reality (MR) scene by configuring a metadata structure, the metadata structure including a series of time information identifications performed on the client device and the server device when processing the received interaction; the server receiving an interaction request having interaction information, the interaction information comprising first time information in a first stage of processing the interaction request on the client device; the server identifying further time information in a second stage of processing the interaction request; and transmitting a response to the interaction request to the client device, the response comprising further interaction information comprising the interaction information and further time information from the server according to the metadata structure, and one or more quality of experience (QoE) metrics being at least partially based on the further interaction information.

[0021]

[0021] In some embodiments of the methods of the additional embodiments, at least one of the first time information and further time information may include one or more timestamps.

[0022]

[0022] Some embodiments of the additional embodiment method may further include receiving an identifier code and identifying the configuration of a metadata structure based on the received identifier code.

[0023]

[0023] Some embodiments of the additional embodiment method may further include receiving an identifier code and using the identifier code to check the processing order of user interactions on the server.

[0024]

[0024] Some embodiments of the additional embodiment method may further include receiving an identifier code and using the identifier code to identify a user interaction delay.

[0025]

[0025] For some embodiments of the method of additional embodiments, identifying user interaction latency may include receiving the time at which the user performed an interaction associated with an identifier code.

[0026]

[0026] Some embodiments of the method of additional embodiments may further include identifying an interaction request drop rate, which may include the rate at which interaction requests are dropped by the server, and the interaction request drop rate may include a QoE metric.

[0027]

[0027] For some embodiments of the method of additional embodiments, at least one of the one or more QoE metrics may be user interaction latency.

[0028]

[0028] For some embodiments of the method of additional embodiments, the response may include an updated description of the MR scene.

[0029]

[0029] For some embodiments of the method of additional embodiments, the response may include a new description of the MR scene.

[0030]

[0030] For some embodiments of the method of additional embodiments, the response may include information indicating that an interaction request has been dropped by the server.

[0031]

[0031] For some embodiments of the method of additional embodiments, the response may include an encoded version of the rendered MR scene.

[0032]

[0032] For some embodiments of the method of additional embodiments, further time information may further include the stage of processing at the server.

[0033]

[0033] Some embodiments of the methods of the additional embodiments may further include receiving one or more quality of experience (QoE) metrics that are at least partially based on further interaction information.

[0034]

[0034] Apparatuses of additional embodiments according to some embodiments may include a processor and a non-temporary computer-readable medium that stores instructions that, when executed by the processor, are operable to cause the apparatus to perform any one of the claims listed above. [Brief explanation of the drawing]

[0035] Brief explanation of the drawing [Figure 1A]

[0035] This is a schematic side view showing one embodiment of a waveguide display that may be used with an augmented reality (XR) application according to several embodiments. [Figure 1B]

[0036] This is a schematic side view showing an alternative display type in one embodiment, which may be used with an augmented reality application according to several embodiments. [Figure 1C]

[0037] This is a schematic side view showing an alternative display type in one embodiment, which may be used with an augmented reality application according to several embodiments. [Figure 1D]

[0038] This is a system diagram showing an interface set for one embodiment of the system, according to several embodiments. [Figure 2]

[0039] This is a system diagram showing one embodiment of an interface set for an MPEG-I node hierarchy that supports elements of scene interactivity, according to several embodiments. [Figure 3]

[0040] This is a system diagram showing an interface set of one embodiment for a standalone augmented reality (STAR)-based 5G interactive immersive service, according to several embodiments. [Figure 4]

[0041] This is a system diagram showing an interface set of one embodiment for an edge-dependent AR (EDGAR)-based 5G interactive immersive service, based on several embodiments. [Figure 5]

[0042] This is a message sequence diagram illustrating one embodiment of the process for user interaction with a standalone XR device, according to several embodiments. [Figure 6]

[0043] This message sequence diagram illustrates one embodiment of the process for user interaction with an XR edge server using split rendering, according to several embodiments. [Figure 7]

[0044] This flowchart illustrates one embodiment of the process for identifying perceived quality metrics, based on several embodiments. [Modes for carrying out the invention]

[0036]

[0045] The entities, connections, structures, etc., shown in and described in relation to the various diagrams are presented as examples, not as limitations. Therefore, any and all descriptions or other references relating to what a particular diagram "depicts," what a particular element or entity in a particular diagram "is" or "has," and any and all similar descriptions that might be read as absolute, and therefore limiting, in isolation and out of context, should only be read appropriately as constructively preceded by clauses such as "in at least one embodiment, ...". For the sake of conciseness and clarity of explanation, this implicit precedence is not repeated at length in the detailed explanation.

[0037] Detailed explanation

[0046] Figure 1A is a schematic side view showing one embodiment of a waveguide display, which may be used in conjunction with an augmented reality (XR) application, according to several embodiments. The image is projected by an image generator 102. The image generator 102 may use one or more of the various techniques for projecting the image. For example, the image generator 102 may be a laser beam scanning (LBS) projector, a liquid crystal display (LCD), a light-emitting diode (LED) display (including organic LED (OLED) or micro-LED (μLED) display), a digital light processor (DLP), a reflective liquid crystal on silicon (LCoS) display, or other types of image generators or light engines.

[0038]

[0047] The light representing the image 112 generated by the image generator 102 is coupled to the waveguide 104 by the diffraction in-coupler 106. The in-coupler 106 diffracts the light representing the image 112 to one or more diffraction orders. For example, ray 108, one of the rays representing a portion of the bottom of the image, is diffracted by the in-coupler 106, and one of the diffraction orders 110 (e.g., the second order) is at an angle at which it can propagate through the waveguide 104 by total internal reflection. The image generator 102 displays the image as instructed by the control module 124, which operates to render image data, video data, point cloud data, or other displayable data.

[0039]

[0048] At least a portion of the light 110 coupled to the waveguide 104 by the diffraction in-coupler 106 is coupled out of the waveguide by the diffraction out-coupler 114. At least a portion of the light coupled out of waveguide 104 reproduces the incident angle of the light coupled to the waveguide. For example, in the figure, the output-coupled rays 116a, 116b, and 116c reproduce the angle of the input-coupled ray 108. Since the light exiting the out-coupler reproduces the direction of the light entering the in-coupler, the waveguide substantially reproduces the original image 112. The user's eye 118 can focus on the reproduced image.

[0040]

[0049] In the embodiment shown in Figure 1A, the outcoupler 114 outputs only a portion of the light, and each reflection allows a single input beam (e.g., beam 108) to generate multiple parallel output beams (e.g., beams 116a, 116b, and 116c). In this way, even if the eye is not perfectly aligned with the center of the outcoupler, at least a portion of the light emanating from each part of the image is likely to reach the user's eye. For example, if the eye 118 attempts to move downward, beam 116c may enter the eye even without beams 116a and 116b, so the user can still perceive the bottom of the image 112 even as their position shifts. The outcoupler 114 therefore partially acts as a vertical exit pupil expander. The waveguide may also include one or more additional exit pupil expanders (not shown in Figure 1A) that expand the exit pupil horizontally.

[0041]

[0050] In some embodiments, the waveguide 104 is at least partially transparent to light generated outside the waveguide display. For example, at least a portion of the light 120 from a real-world object (e.g., object 122) crosses the waveguide 104, allowing the user to see the real-world object while using the waveguide display. Since the light 120 from the real-world object also passes through the diffraction grating 114, multiple diffraction orders and therefore multiple images exist. To minimize the visibility of multiple images, it is desirable that diffraction order zero (no deviation by 114) has a high diffraction efficiency for light 120 and order zero, while higher diffraction orders have lower energy. Therefore, in addition to scaling and output coupling the virtual image, the outcoupler 114 is preferably configured to allow the zeroth order of the real image to pass through. In such embodiments, the image displayed by the waveguide display may appear superimposed on the real world.

[0042]

[0051] Figure 1B is a schematic side view showing an alternative display type in one embodiment, which may be used with an augmented reality application, according to several embodiments. In the XR head-mounted display device 130, a control module 132 controls a display 134, which may be an LCD, that displays an image. The head-mounted display includes a partial reflective surface 136 that reflects (and, in some embodiments, both reflects and focuses) the image displayed on the LCD so that the image is visible to the user. The partial reflective surface 136 also allows at least some ambient light to pass through, allowing the user to see their surroundings.

[0043]

[0052] Figure 1C is a schematic side view showing an alternative display type in one embodiment, which may be used with an augmented reality application, according to several embodiments. In the XR head-mounted display device 140, a control module 142 controls a display 144, which may be an LCD, that displays images. The images are focused by one or more lenses of the display optics 146 to make the images visible to the user. In the embodiment of Figure 1C, ambient light does not directly reach the user's eyes. However, in some such embodiments, an external camera 148 may be used to capture images of the external environment, and such images may be displayed on the display 144 along with any virtual content, which may also be displayed.

[0044]

[0053] The embodiments described herein are not limited to any particular type or structure of XR display device.

[0045]

[0054] Figure 1D is a system diagram showing an interface set of one embodiment for a system, according to several embodiments. The augmented reality display device, together with its control electronics, may be implemented using a system such as the system in Figure 1D. System 150 can be embodied as a device including various components described below and configured to perform one or more of the embodiments described in this document. Embodiments of such a device include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected consumer electronics, and servers. The elements of System 150 can be embodied individually or in combination as a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of System 150 are distributed across multiple ICs and / or discrete components. In various embodiments, System 150 is communicably coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, the system 1000 is configured to implement one or more of the embodiments described in this document.

[0046]

[0055] System 150 includes, for example, at least one processor 152 configured to execute loaded instructions in order to implement various embodiments described in this document. The processor 152 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 150 includes at least one memory 154 (e.g., a volatile memory device and / or a non-volatile memory device). System 150 may also include a storage device 158 that may contain non-volatile memory and / or volatile memory, and includes, but is not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drives, and / or optical disk drives. The storage device 158 may, in non-limiting examples, include an internal storage device, a mounted storage device (including removable and non-removable storage devices), and / or a network-accessible storage device.

[0047]

[0056] System 150 includes, for example, an encoder / decoder module 156 configured to process data and provide encoded or decoded video, the encoder / decoder module 156 of which may include its own processor and memory. The encoder / decoder module 156 represents a module that can be included in a device that performs encoding and / or decoding functions. As is well known, the device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 156 may be implemented as a separate element of System 150 or may be incorporated into the processor 152 as a combination of hardware and software, as is known to those skilled in the art.

[0048]

[0057] Program code loaded onto the processor 152 or encoder / decoder 156 to perform various embodiments described in this document can be stored in the storage device 158 and then loaded into memory 154 for execution by the processor 152. According to various embodiments, one or more of the processor 152, memory 154, storage device 158, and encoder / decoder module 156 can store one or more of various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of mathematical formulas, equations, operations, and operational logic.

[0049]

[0058] In some embodiments, memory within the processor 152 and / or encoder / decoder module 156 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory outside the processing device (for example, the processing device can be either the processor 152 or the encoder / decoder module 152) is used for one or more of these functions. The external memory can be memory 154 and / or storage device 158, for example, dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used, for example, to store the television's operating system. In at least one embodiment, high-speed external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, for example, with respect to MPEG-2 (MPEG refers to the Moving Picture Experts Group, also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET (the Joint Video Experts Team)).

[0050]

[0059] Inputs to the elements of System 150 can be provided via various input devices, as shown in Block 172. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion for receiving RF signals transmitted by broadcast by a broadcasting station, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other embodiments, not shown in Figure 1C, include composite video.

[0051]

[0060] In various embodiments, the input device of block 172 has associated input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also known as selecting a signal or band-limiting a signal to a frequency in a certain band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal frequency band that can be called a channel in a particular embodiment, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF portion of various embodiments may include one or more elements for performing these functions, e.g., a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner for performing various of these functions, e.g., down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency near the baseband) or to the baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive an RF signal transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments may rearrange the order of the elements described above (and others), remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF section includes an antenna.

[0052]

[0061] In addition, the USB and / or HDMI terminals may include their respective interface processors for connecting the system 150 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, can be implemented, for example, in a separate input processing IC or within processor 152, as needed. Similarly, aspects of USB or HDMI interface processing can be implemented, for example, within a separate interface IC or within processor 152, as needed. Demodulation, error correction, and multiplexed streams are provided to various processing elements, such as processor 152 and encoder / decoder 156 operating in conjunction with memory and storage elements, to process the data stream for presentation to an output device, as needed.

[0053]

[0062] Various elements of system 150 can be housed within an integrated housing. Within the integrated housing, the various elements are interconnected using an appropriate connection configuration 174, such as an internal bus known in the art, including an Inter-IC (I2C) bus, wiring, and a printed circuit board, and data can be transmitted between these elements.

[0054]

[0063] System 150 includes a communication interface 160 that enables communication with other devices via a communication channel 162. The communication interface 160 may include, but is not limited to, transceivers configured to transmit and receive data via the communication channel 162. The communication interface 160 may include, but is not limited to, a modem or a network card, and the communication channel 162 may be implemented, for example, within a wired and / or wireless medium.

[0055]

[0064] In various embodiments, data is streamed to system 150 or provided in other ways using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 162 and a communication interface 160 adapted for Wi-Fi communication. In these embodiments, the communication channel 162 is typically connected to an access point or router that provides access to an external network, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, the streamed data is provided to system 150 using a set-top box that transmits data via an HDMI connection in input block 172. In yet another embodiment, the streamed data is provided to system 150 using an RF connection in input block 172. As described above, various embodiments provide data in non-streaming manners. In addition, various embodiments use wireless networks other than Wi-Fi, e.g., cellular networks or Bluetooth networks.

[0056]

[0065] System 150 can provide output signals to various output devices, including a display 176, a speaker 178, and other peripheral devices 180. In various embodiments, the display 176 includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 176 may be for a television, tablet, laptop, mobile phone, or other device. The display 176 may also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop). In various examples of embodiments, the other peripheral devices 180 include one or more of a standalone digital video disc (or digital multipurpose disc) (both terms refer to DVD), a disc player, a stereo system, and / or a lighting system. Various embodiments utilize one or more peripheral devices 180 that provide functions based on the output of System 150. For example, a disc player performs the function of playing back the output of System 150.

[0057]

[0066] In various embodiments, control signals are transmitted between the system 150 and the display 176, speaker 178, or other peripheral devices 180 using signal transmission such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable control between devices with or without user intervention. Output devices can be communicatively coupled to the system 1000 via dedicated connections through their respective interfaces 164, 166, and 168. Alternatively, output devices can be connected to the system 150 using a communication channel 162 via a communication interface 160. The display 176 and speaker 178 can be integrated into a single unit with other components of the system 150 in an electronic device such as a television. In various embodiments, the display interface 164 includes a display driver, such as a timing controller (TCon) chip.

[0058]

[0067] The display 176 and speaker 178 can, alternatively, be isolated from one or more of the other components, for example, if the RF portion of input 172 is part of a separate set-top box. In various embodiments where the display 176 and speaker 178 are external components, the output signal can be provided via a dedicated output connection, such as an HDMI port, a USB port, or a COMP output.

[0059]

[0068] System 150 may include one or more sensor devices 168. Examples of sensor devices that may be used include one or more GPS sensors, gyroscopes, accelerometers, optical sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors may be used to determine information such as the user's position and orientation. If System 150 is used as a control module for an augmented reality display (control modules 124, 132, etc.), the user's position and orientation may be used to determine how to render image data so that the user perceives the correct parts of virtual objects or virtual scenes from the correct viewpoint. In the case of a head-mounted display device, the position and orientation of the device itself may be used to determine the user's position and orientation for the purpose of rendering virtual content. In the case of other display devices such as telephones, tablets, computer monitors, or televisions, other inputs may be used to determine the user's position and orientation for the purpose of rendering content. For example, the user may select and / or adjust a desired viewpoint and / or gaze direction by using a touchscreen, keypad or keyboard, trackball, joystick, or other input. If the display device has sensors such as an accelerometer and / or gyroscope, the viewpoint and orientation used for rendering content may be selected and / or adjusted based on the movement of the display device.

[0060]

[0069] The embodiments can be executed by the processor 152 or other hardware, or by computer software implemented by a combination of hardware and software. In a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 154 can be of any type appropriate to the technical environment and, in a non-limiting example, can be implemented using any appropriate data storage technology such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 152 can be of any type appropriate to the technical environment and, in a non-limiting example, can include one or more of a microprocessor, a general-purpose computer, a dedicated computer, and a processor based on a multi-core architecture.

[0061] Scene description framework for XR

[0070] This principle generally relates to the rendering of augmented reality scene descriptions and augmented reality rendering. This document is also understood in relation to formats and playback of augmented reality applications when rendered on end-user devices such as mobile devices or head-mounted displays (HMDs).

[0062]

[0071] Augmented reality (XR) is a technology that enables interactive experiences in which real-world environments and / or video content are enhanced by virtual content that can be defined across multiple sensory modalities, including sight, hearing, and touch. When an application is executed, virtual content (e.g., 3D content or audio / video files) is rendered in real time in a manner consistent with the user context (environment, viewpoint, device, etc.). Scene graphs (e.g., those proposed by Khronos / glTF, and their extensions defined in the MPEG Scene Description format or Apple / USDZ) are possible ways of representing the rendered content. They combine, on the one hand, a declarative description of the scene structure that links real-world objects and virtual objects, and on the other hand, a binary representation of the virtual content.

[0063]

[0072] While such a scene description framework ensures that time-sensitive media and their corresponding associated virtual content are always available during application rendering, it does not describe how users can interact with scene objects at runtime for an immersive XR experience.

[0064]

[0073] No XR system can take an XR scene description that includes metadata describing how a user can interact with scene objects at runtime, and how these interactions may be updated when an XR application is running.

[0065]

[0074] In XR applications, scene descriptions are used to combine an explicit and easily analyzable description of the scene structure with a binary representation of the media content.

[0066]

[0075] In time-based media streaming, the scene description itself may evolve over time to provide virtual content related to each sequence of the media stream. For example, for advertising purposes, a virtual bottle may be displayed during a video sequence in which people are drinking.

[0067]

[0076] In some embodiments, the frameworks described in the literature, Scene Description for MPEG Media Document, Information Technology-Coded Representation of Immersive Media - Part 14: Scene Description for MPEG Media, International Organization for Standardization (ISO), and ISO / IEC DIS 23090-14:2021(E)(2021) may be used.

[0068] Runtime interactivity

[0077] Figure 2 is a system diagram showing an embodiment of the interface set for an MPEG-I node hierarchy supporting elements of scene interactivity, according to several embodiments. According to this principle, some are shown in node hierarchy 200, and in addition to the node tree, behavior metadata items (embodied in this specification referred to as “behavior”) are added to the scene description. In one embodiment, a time-evolving scene description is enhanced by adding information that identifies behavior. These behaviors may relate to predefined virtual objects that allow runtime interactivity for user-specific XR experiences.

[0069]

[0078] In some embodiments, these behaviors are time-evolving. In such embodiments, the behaviors may be updated via an existing scene description update mechanism.

[0070]

[0079] In one example embodiment, the behavior may be characterized by one or more of the following characteristics: • One or more triggers that define the conditions that must be met for activation. • Trigger control parameters that define logical operations between defined triggers. • Actions implemented in response to trigger activation. • Action control parameters that define the execution order of defined actions. • A priority number that allows the highest priority behavior to be selected when multiple behaviors are running concurrently on the same virtual object. • An interrupt action that specifies how to terminate a behavior when it is no longer defined in a newly received scene update. For example, the behavior becomes undefined if the associated object is deleted or is no longer relevant to the current media (e.g., audio or video) sequence.

[0071]

[0080] By adding these behaviors, time-dependent user interactivity in immersive content for XR experiences may be defined.

[0072]

[0081] When the second scene description is received, some of the behaviors of the first scene description may become "in progress," for example, they may be triggered and their actions may be executed. The second scene description may be provided as update metadata, for example, metadata describing the differences between the first and second scene descriptions. The second scene description includes a node tree that describes objects which may be common to or different from the objects of the first scene description. Objects in the node tree of the first scene description may no longer exist in the second description. If objects relating to the execution actions of the in-progress behaviors are missing in the second scene description, those in-progress behaviors are no longer applicable. If the in-progress behaviors are not defined in the second description, the in-progress behaviors are no longer applicable. The interrupt action field describes how to interrupt the in-progress behaviors with execution actions.

[0073]

[0082] In XR applications, scene descriptions are used to combine an explicit and easily analyzable description of the scene structure with a binary representation of the media content.

[0074]

[0083] In this application, the user equipment (UE) may correspond to any augmented reality (XR) device / node, which may be of various form factors. A typical UE (e.g., an XR UE) may include, but is not limited to, the following: head-mounted displays (HMDs), optical see-through glasses and camera see-through HMDs for AR and MR, mobile devices with position tracking and cameras, and wearables. In addition to the above, several different types of XR UEs, e.g., displays, cameras, sensors, sensor processing, wireless connectivity, XR / media processing, and power supply devices, may be used based on the XR device capabilities and may be provided by one or more devices, wearables, actuators, controllers, and / or accessories. One or more devices / nodes / UEs may be grouped into a collaborative XR cluster to support an XR application / experience / service.

[0075] System Architecture

[0084] In the specification for SUPPORT OF 5G GLASS-TYPE AUGMENTED REALITY / MIXED REALITY (AR / MR) DEVICES, 3rd Generation Partnership Project (3GPP), TR26.998, Release 17.1.0 (Sep. 2022), the 3GPP Technical Specification Group Services and System Aspects Working Group 4 (SA4) defines the system architecture for interactive immersive services. Different degrees of partitioned workflow between AR devices and the Cloud / Edge are identified, resulting in the definition of two main device types: 5G Standalone AR UE (STAR) and 5G Edge-Dependent AR UE (EDGAR).

[0076] 5G Standalone AR UE(STAR)

[0085] Figure 3 is a system diagram showing an interface set of one embodiment for a standalone augmented reality (STAR)-based 5G interactive immersive service in several embodiments. Figure 3 provides an architecture 300 for immersive interactive media delivery using STAR UE. The left side of Figure 3 is a UE device 302 which may function as an XR client in several embodiments. The right side of Figure 3 shows an XR server 304, and the UE 302 and XR server 304 may be connected by a network 306 shown in the center of Figure 3. Inside the UE / XR client, an XR application 308 may run. The XR application 308 may have bidirectional access to a network interface 310, a media access function 312, and an XR source management block 314. The XR application 308 may also interface to an XR runtime block 316 and a scene manager block 318.

[0077]

[0086] In some embodiments, the XR runtime block 316 may interface with one or more sensors 320, one or more cameras 322, one or more actuators 324, one or more displays 326, one or more speakers 328, one or more user interfaces (such as a pad controller), one or more lights, and one or more buttons. In some embodiments, the XR runtime 316 may receive information from the presentation engine 330 and / or the scene manager 318. The presentation engine 330 may include a renderer block 332.

[0078]

[0087] In some embodiments, user actions or other inputs may be received by the XR runtime block 316. Such user actions or other inputs may be passed to the XR source control block 314, then to the media access function (MAF) 312, and further to the network block 310 for transmission to the XR server 304. In some embodiments, interaction responses or other messages may be received by the network block 310. Such information may be passed to the media access function (MAF) 312 and the presentation engine 330.

[0079]

[0088] An XR application 334 may be executed within the XR server 304. The XR application 334 may have bidirectional access to the network interface 336 and the media delivery function 338. Within the XR application 334 running on the XR server 304, an XR function 340 and a scene manager 342 may be executed. The XR application 334 may also have access to media assets 344. Within the scene manager block 342, a scene graph handler 346 may exist. The media delivery function 338 for the XR server 305 may have bidirectional access to the network interface 336 and the XR application 334.

[0080] 5G edge-dependent AR UE (EDGAR)

[0089] Figure 4 is a system diagram showing an interface set of one embodiment for an edge-dependent AR (EDGAR) based 5G interactive immersive service, according to several embodiments. Figure 4 provides an architecture 400 for interactive immersive media delivery using EDGAR UE 402. In this case, most of the rendering is performed on server 404.

[0081]

[0090] Comparing Figures 3 and 4, most of the interfaces are the same, but there are some differences. For information on the same items, please refer to the explanation of Figure 3. Regarding the differences, Figure 3 shows the renderer block 332 inside the UE presentation engine block 330. In Figure 4, the server's scene manager block 406 shows the immersive renderer block 408. In addition, Figure 3 shows the media asset 344 inside the XR application block 334 of the XR server, while Figure 4 shows a bidirectional interface between the XR application provider 410, which has the media asset 412, and the XR application block 414.

[0082] Interactive immersive service

[0091] In some embodiments, such as shared interactive immersive services, user interactions are sent from the UE to the server. User interactions may be asynchronous with other data flows or may be single events. Furthermore, the frequency of interaction events depends on the type of interaction and the use case. The server processes user requests to the immersive media scene (e.g., by changing the context such as translating, rotating, and / or scaling, or by adding new objects to the scene).

[0083]

[0092] In a UE standalone configuration, UE can render the scene, and the server sends the processed scene to UE. In some embodiments, the processed scene is sent as a scene description update file. In some embodiments, the processed scene is sent as a completely new scene description.

[0084]

[0093] In an edge-assisted UE configuration, which may also be called split rendering, UE offloads scene rendering to a server. The server rasterizes the XR viewport, pre-renders it to generate XR media, encodes it, and sends it back to UE.

[0085] Interactivity delay and QoE

[0094] In relation to interactive immersive services, round-trip interaction delay may be used to evaluate user experience quality, as defined in the document EXTENDED REALITY (XR) IN 5G, 3rd Generation Partnership Project (3GPP), TR26.928, Release 17.0.0 (April 2022) ("3GPP TR26.928") (see Section 4.2.2, Interaction Delays and Age of Content). In some embodiments, round-trip interaction delay is an important parameter for estimating user experience quality. Round-trip interaction delay is the sum of user interaction delay and content lifetime. In some embodiments, round-trip interaction delay itself may be used as the quality of experience (QoE) metric.

[0086]

[0095] User interaction delay is the time between when a user action is initiated and when that action is processed and considered by the content creation engine. User interaction delay can be affected by wireless network uplink delay.

[0087]

[0096] The content's lifespan is the period between when the content is created and when it is presented to the user. The content's lifespan may be affected by wireless network downlink latency.

[0088]

[0097] Figures 5 and 6, which will be discussed later in this application, both show user interaction delay, content validity period, and round-trip interaction delay.

[0089]

[0098] For some embodiments, interactivity quality (QoE) is highly dependent on round-trip interaction delay and use case. Table 4.2.2-1 of 3GPP TR26.928, reproduced here as Table 1, provides, for example, acceptable round-trip interaction delays for different game types.

[0090] [Table 1]

[0091]

[0099] In some embodiments, content latency is specified, for example, by a conversational latency threshold. Typically, latency of approximately 200 milliseconds is acceptable. Overall, different applications and use cases have different latency requirements.

[0092]

[0100] Table 2 lists four categories of interactions / applications related to round-trip interaction delays.

[0093] [Table 2]

[0094]

[0101] Multiple interactions with different round-trip interaction delay thresholds may coexist within an application.

[0095] openXR interaction

[0102] registry<dot>chronos <dot>org / openxr / specs / 1.0 / html / xrspec <dot>html The standard OpenXR specification available in #input, the Khronos OpenXR Working Group (version 1.0.27) ("Khronos OpenXR"), provides an API for accessing VR / AR platforms and devices. One of the elements of this API is the XrActions element, which is used to handle user interactions.

[0096]

[0103] The current state of an input action may also be obtained by calling the xrGetActionState* function, which returns an XrActionState* structure containing the lastChangeTime timestamp of the last change to the state of this action. The lastChangeTime timestamp corresponds to the time the user performed the action.

[0097]

[0104] Estimating interactivity QoE properties is related to measuring round-trip interaction delay. While the openXR API exposes the time it takes for a user to perform an action, UE applications lack a way to know when this action is considered by the server and when the server receives the interaction response, due to a lack of predefined procedures.

[0098]

[0105] It is important to monitor the contribution of various steps to the overall delay through the wireless network, UE, and server process blocks. Different delay contributions from the processing pipeline may include processing time in the UE and server, as well as transmission time through the wireless network for both uplink and downlink. Knowing this information, applications may make several adjustments through the processing pipeline to optimize resources in the UE, network (uplink and downlink), and server / edge for the target interactivity QoE.

[0099]

[0106] In some embodiments, such as within the context of 5G applications, different latency monitoring may include specifying that timing metadata exchanged between the UE and the server may include interactivity QoE metrics that may be shared with the 5G network.

[0100]

[0107] This application considers adding timestamp and identifier metadata to interaction requests from user devices (UEs) to edge / servers and interaction responses from edge / servers to UEs. UEs and servers may exchange interaction requests and responses with interactivity QoE metadata. At each step of the process, UEs and edge / server applications may record time and transmit the time along with the interaction data. Such functionality allows for the measurement of delay at each step across the network and the estimation of each step's contribution to the perceived quality of user interactivity. Measuring the temporal variation of different delays may help optimize steps, such as identifying the step with the greatest delay variation, and improve efficiency. The server replicates the interactivity QoE metadata received from the UE and attaches the metadata to process output messages sent back to the UE. The persistence of interactivity QoE metadata through the interaction processing pipeline may help in measuring delay.

[0101] Interaction QoE configuration

[0108] The configuration of the interaction QoE may occur between the UE and the server during application initialization, which may include setting up one or more application sessions. Such configuration may include setting where, how, and in what form the interaction QoE will be measured and reported.

[0102]

[0109] The UE and server negotiate the configuration of the interaction QoE for each interaction specified in the application. For example, the configuration may include which interaction timestamps are recorded throughout the interaction processing pipeline. The granularity of this timestamp recording may depend on several factors, such as the interaction categories shown in Table 2 above, and / or the frequency of the interactions.

[0103] Measuring delay time in the STAR architecture

[0110] Figure 5 is a message sequence diagram illustrating the process of one embodiment for user interaction with a standalone XR device, according to several embodiments. For 5G standalone AR (STAR) device types, the scene manager on the server maintains a consistent scene across multiple users. The server scene manager 526 periodically sends the scene state to the UE presentation engine 530, which updates the scene graph and renders the scene according to the latest user pose.

[0104]

[0111] Figure 5 shows step 500 of one embodiment for the interactivity timeline / pipeline. In some embodiments, the XR application on both UE522 and server 524 has already been initialized, and all sessions have been established and configured. Figure 5 does not show user pause and other sensor timelines / pipelines, but such actions may still occur. The following process 500 corresponds to Figure 5.

[0105]

[0112] UE522 and server 524 constitute interaction QoE 502.

[0106]

[0113] The raw user action 504 is retrieved from the XR runtime 534 by XR source control 532.

[0107]

[0114] XR source control 532 formats the raw user actions into interaction information, which includes interactivity QoE metadata. A unique identifier for the user-action-time stamp and interaction user-action-id is added to the interactivity QoE metadata. The interaction information 506 is shared with the Media Access Function (MAF) 528.

[0108]

[0115] MAF528 sends interaction request 508 to scene manager 526 in XR server 524. MAF528 appends the action-req-TX-time timestamp to interactivity QoE metadata.

[0109]

[0116] Interaction requests are received by the XR server 524 and buffered before being processed during the next iteration of the update loop in the scene manager 526. The scene manager 526 in the server 524 processes interaction tasks according to the interaction requests from UE 522 and updates the scene. The scene manager 526 records a scene-update-time timestamp in the interactivity QoE metadata when the scene manager 526 begins to process the interaction request 510. Using the information in the interactivity QoE metadata, the server application may calculate equations 1 and 2. User Interaction Delay = Scene Update Time - User Action Time(1) Tx Action Delay = Scene Update Time - Action Request Tx Time(2) Scene Manager 526 may ignore interaction request 508 according to application policies, namely, too many requests, requests that are too late, or low priority. In this case, interactivity QoE metadata, including the user-action-id of the dropped interaction, may be appended to the interactivity QoE metadata of the next interaction response to notify the UE. If the interaction was processed by the Scene Manager, the Interaction Complete field is set to TRUE; otherwise, the Integration Complete field is set to FALSE.

[0110]

[0117] Depending on the level of processing, the interaction response 512 may be a scene description update or a new scene description. The interaction response may provide user feedback regarding the interaction task (e.g., virtual hand, virtual ray, sound, and haptic feedback). The interaction response is sent to the UE presentation engine 530. The interaction response includes interactivity QoE metadata from the interaction request, and if the response is sent, it is timestamped, namely scene-update-time and interaction-resp-TX-time timestamps.

[0111]

[0118] When the UE MAF528 receives an interaction response, the UE application records the interaction-resp-RX-time timestamp associated with the interaction response and extracts the interactivity QoE. The transmission time over the downlink radio network may be estimated using interaction-resp-TX-time. The presentation engine 530 updates the scene graph 514 or loads a new scene in response to the interaction request. The presentation engine 530 renders the scene in the next iteration of the update loop. The start-render-time, when the scene begins rendering, is captured by the application and appended to the interactivity QoE metadata.

[0112]

[0119] The rendered frame 516 is shared with the XR runtime 534.

[0113]

[0120] The XR runtime 534 performs further post-processing 518 before presenting it to the user.

[0114]

[0121] The rendered frame with interaction response is presented to the user via the display, speaker, and / or actuator 520. The application captures the presentation time.

[0115] Measuring delay time in the EDGAR architecture

[0122] Figure 6 is a message sequence diagram illustrating the process of one embodiment for user interaction with an XR edge server using split rendering, according to several embodiments.

[0116]

[0123] For 5G edge-dependent AR (EDGAR) device types, the server-side scene manager maintains a consistent scene across multiple users. The server scene manager is responsible for pre-rendering the scene for the UE using the most recent user pose. The server scene manager also encodes the rendered frames and sends the encoded frames back to the UE. The UE decodes the rendered frames, performs further post-processing such as pose correction, and presents the frames to the user.

[0117]

[0124] Figure 6 shows the procedure for one embodiment of the interactivity timeline / pipeline. In some embodiments, the XR application on both the UE and the server is already initialized, and all sessions are established and configured. Figure 6 does not show user pause and other sensor timelines / pipelines, but such actions may still occur. The following process 600 corresponds to Figure 6.

[0118]

[0125] UE632 and server 634 constitute 602, which is the interaction QoE.

[0119]

[0126] The raw user action 604 is retrieved from the XR runtime 648 by XR source control 646.

[0120]

[0127] XR source control 646 formats the raw user action 604 into interaction information that includes interactivity QoE metadata. A user-action-time stamp and a unique identifier, user-action-id, are added to the interactivity QoE metadata. The interaction information 606 is shared with the Media Access Function (MAF) 642.

[0121]

[0128] MAF642 sends interaction request 608 to scene manager 636 in XR server 634. MAF642 appends the action-req-TX-time timestamp to the interactivity QoE metadata.

[0122]

[0129] Interaction request 608 is received by the XR server 634 and buffered before being processed during the next iteration of the update loop of the scene manager 636. The scene manager 636 on the server processes the interaction task according to the interaction request from UE632 and updates the scene. The scene manager 636 records a scene-update-time timestamp when it begins to process the interaction request 610. Using the information in the interactivity QoE metadata, the server application may calculate equations 3 and 4. User Interaction Delay = Scene Update Time - User Action Time Expression 3 Tx Action Delay = Scene Update Time - Action Request Tx Time Equation 4 As shown in the STAR architecture (Figure 5), the scene manager 636 may ignore the interaction request 608 according to the application policy. If the interaction is handled by the scene manager, the interaction complete field is set to TRUE; otherwise, the integration complete field is set to FALSE.

[0123]

[0130] The scene manager 636 shares the scene state 612 with the renderer 638 in the server 634.

[0124]

[0131] The scene is rendered using the last predicted user pose. The 614 when the scene begins rendering and the start-render-time are captured by the application and stored in the interactivity QoE metadata.

[0125]

[0132] The rendered media frame 618 is shared with the media output function 640.

[0126]

[0133] The server media output function 640 encodes the rendered media frame 620. The server media output function 640 records the frame-encode-time when it starts encoding the frame 620 and appends this timestamp to the interactivity QoE metadata associated with the media frame.

[0127]

[0134] The encoded media frame 622 is sent from server 640 to UE MAF642 along with interactivity QoE metadata. When the server sends the rendered media frame to UE632, it records the rendered-frame-TX-time timestamp and appends the rendered-frame-TX-time timestamp to the interactivity QoE metadata.

[0128]

[0135] UE MAF642 receives and ingests the rendered-frame-TX-time within the interactivity QoE metadata, and then MAF642 decodes the rendered media frame.

[0129]

[0136] The rendered frame 626 is shared with the presentation engine 644 and the XR runtime 648. The UE application takes the frame-decoded-time in the interactivity QoE metadata.

[0130]

[0137] The XR runtime 648 performs further post-processing 628, such as pose correction, before presenting the frame to the user.

[0131]

[0138] The rendered frame with interaction response is presented to the user via the display, speaker, and / or actuator 630. The application captures the presentation time.

[0132] Interactivity QoE processing model:

[0139] Using interactivity QoE metadata from the server and timestamps within the UE, the UE application may, for example, calculate equations 5-8, which include: User Interaction Delay = Scene Update Time - User Action Time Formula 5 Tx Action Delay = Scene Update Time - Action Request Tx Time Equation 6 Age of Content = Presentation Time - Scene Update Time Equation 7 Roundtrip Interaction Delay = Presentation Time - User Action Time Formula 8

[0133]

[0140] In some embodiments, a UE application may calculate the round-trip interaction delay by using an interaction identifier, user-action-id, to store the time the user performed an interaction with this identifier. Using the interaction identifier, user-action-id, in the interactivity QoE response, the UE application may determine whether the user interaction was considered by the server or dropped / ignored. The drop rate may be used as a QoE metric. The interaction identifier, user-action-id, may also be used to check the order in which interactions are processed by the server.

[0134]

[0141] Some timestamps and delays may not be available depending on the granularity of timestamp recording specified in the configuration. For example, for certain types of interactions, the application may be configured not to record the start rendering time. For this type of interaction, the server may measure the scene update time against the frame encoding time delay corresponding to the interaction processing delay plus the scene rendering delay. In this case, the interaction processing delay and the scene rendering delay may not need to be measured.

[0135]

[0142] The UE reports the resulting interactivity QoE metadata to the server along with all round-trip interaction timestamps. The UE and server application may use this report to optimize user QoE for subsequent user interactions.

[0136]

[0143] Fine-grained optimization of the interaction pipeline's network and computing resources may be performed using timestamps within the interactivity QoE metadata. This optimization may depend on the interaction / application category related to the round-trip interaction delay threshold described in Table 2.

[0137]

[0144] For example, if an interaction falls into a non-critical latency category, wireless network latency for the uplink may be mitigated to free up some wireless resources for critical data flows with other latency. The application reports the interaction QoE to the edge server, and QoE-aware edge resource orchestration may allocate computing power to other processes.

[0138] Interactivity QoE Metadata Format

[0145] Interactivity QoE metadata may consist of multiple fields that depend on the system architecture. Standalone (star-shaped) or edge-dependent (EDGAR) device types have specific delays and timestamps, but there are also delays and timestamps common to both configurations. The item interactionQoESets is the name of the container that holds each of the elements shown in Table 3. In some embodiments, the interactivity QoE metadata structure may be, for example, in JSON format and may follow the syntax and semantics shown in Table 3.

[0139] [Table 3]

[0140] [Table 4]

[0141] transfer

[0146] In some embodiments, when interactivity QoE metadata is associated with a rendered media frame in a media stream (video or audio) over the RTP, typically due to the type of split rendering device in the downlink, the interactivity QoE metadata may be carried using an RTP header extension.

[0142]

[0147] In some embodiments, interactivity QoE metadata may be sent with the interaction request or response using a WebRTC data channel when no associated media stream is available.

[0143]

[0148] Aspects of the present invention may be detected via the specific semantics described below. ·MEDIA CAPABILITIES FOR AUGMENTED REALITY (MECAR), 3rd Generation Partnership Project (3GPP), TR26.119, Version 0.1.0 (April 2022), • MECAR PERMANENT DOCUMENT V5.1, 3rd Generation Partnership Project (3GPP), Version 5.1, available at www <dot>3gpp <dot>org / ftp / tsg_sa / wg4_codec / 3gpp_sa4_ahoc_mtgs / sa4_video / docs / s4av230017 <dot>zip (specifications), and / or, • MECAR PERMANENT DOCUMENT V5.1, 3rd Generation Partnership Project (3GPP), Version 5.1, available at www <dot>3gpp <dot>org / ftp / tsg_sa / wg4_codec / 3gpp_sa4_ahoc_mtgs / sa4_video / docs / s4av230017 <dot>zip (diagram).

[0144]

[0149] Figure 7 is a flowchart illustrating one embodiment of the process for identifying a perceived quality metric according to several embodiments. In some embodiments, the process 700 of one embodiment may include a client device communicating with a server to configure a metadata structure to manage the processing of received interactions relating to a mixed reality (MR) scene, the metadata structure comprising a series of time information identifications performed by the client device and the server device when processing received interactions. In some embodiments, the process 700 of one embodiment may further include receiving a user action relating to an MR scene 704 and initiating processing of an interaction request based on the user action according to the interaction information and metadata structure corresponding to the user action. In some embodiments, the process 700 of one embodiment may further include the client device identifying first time information in a first stage of processing the interaction request 706. In some embodiments, the process 700 of one embodiment may further include updating the interaction information with the first time information in a first stage 708. In some embodiments, the process 700 of one embodiment may further include sending the interaction request to the server with the updated interaction information 710. In some embodiments, the process 700 of one embodiment may further include receiving a response from a server to an interaction request 712, the response comprising further interaction information with updated interaction information and further time information from the server according to a metadata structure. In some embodiments, the process 700 of one embodiment may further include identifying one or more quality of experience (QoE) metrics based at least in part on the updated further interaction information 714.

[0145]

[0150] While several embodiments of methods and systems are generally discussed in relation to augmented reality (XR), some embodiments may be applied to any XR context, such as a virtual reality (VR) / mixed reality (MR) / augmented reality (AR) context. Furthermore, the term “head-mounted display (HMD)” is used herein according to some embodiments, but some embodiments may be applied to a wearable device (which may or may not be head-mounted) capable of XR, VR, AR, and / or MR, for example, in some embodiments.

[0146]

[0151] One embodiment of the method, according to several embodiments, involves a client device communicating with a server to configure a metadata structure to manage the processing of received interactions relating to a mixed reality (MR) scene, the metadata structure comprising a series of time information identifications performed by the client device and the server device when processing received interactions; receiving a user action relating to an MR scene and initiating processing of an interaction request based on the user action according to the interaction information and metadata structure corresponding to the user action; the client device identifying first time information in a first stage of processing the interaction request; updating the interaction information with the first time information in the first stage; sending the interaction request to the server along with the updated interaction information; receiving a response from the server to the interaction request, the response comprising further interaction information comprising the updated interaction information and further time information from the server according to the metadata structure; and identifying one or more quality of experience (QoE) metrics at least in part based on the updated further interaction information.

[0147]

[0152] Some embodiments of the method of the embodiment may further include identifying second time information in a second stage of processing interaction requests in a client device, and updating interaction information with the second time information in the second stage.

[0148]

[0153] In some embodiments of the method of the embodiment, at least one of the first time information and the second time information may include one or more timestamps.

[0149]

[0154] Some embodiments of the method of the embodiment may further include, in the client device, identifying second time information in a second stage of processing the response from the server, and updating further interaction information from the response from the server with the second time information in the second stage.

[0150]

[0155] Some embodiments of the method of the embodiment may further include receiving an identifier code and identifying the configuration of a metadata structure based on the received identifier code.

[0151]

[0156] Some embodiments of the method described in the examples may further include receiving an identifier code and using the identifier code to check the processing order of user interactions on the server.

[0152]

[0157] Some embodiments of the method of the examples may further include receiving an identifier code and using the identifier code to identify a round-trip interaction delay.

[0153]

[0158] In some embodiments of the method described in the examples, identifying the round-trip interaction delay may include storing the time the user performed the interaction associated with the identifier code.

[0154]

[0159] Some embodiments of the method of the embodiment may further include determining the interaction request drop rate, the interaction request drop rate may include the rate at which interaction requests are dropped by the server, and the interaction request drop rate may include a QoE metric.

[0155]

[0160] In some embodiments of the method described in the examples, at least one of the one or more QoE metrics may be round-trip interaction delay.

[0156]

[0161] For some embodiments of the method described in the examples, the response may include an updated description of the MR scene.

[0157]

[0162] For some embodiments of the method described in the examples, the response may include a new description of the MR scene.

[0158]

[0163] In some embodiments of the method described in the Examples, the response may include information indicating that the interaction request was dropped by the server.

[0159]

[0164] In some embodiments of the method described in the examples, the response may include an encoded version of the rendered MR scene.

[0160]

[0165] In some embodiments of the method described in the examples, further time information may be included from the processing stage on the server.

[0161]

[0166] An embodiment of the apparatus according to several embodiments may include a processor and a non-temporary computer-readable medium that stores instructions that, when executed by the processor, are operable to cause the apparatus to perform one of the methods listed above.

[0162]

[0167] A method in an additional embodiment, according to several embodiments, may include a server receiving communications from a client device and managing the processing of user interactions relating to a mixed reality (MR) scene by configuring a metadata structure, the metadata structure including a series of time information identifications performed on the client device and server device when processing the received interaction; the server receiving an interaction request having interaction information, the interaction information comprising first time information in a first stage of processing the interaction request on the client device; the server identifying further time information in a second stage of processing the interaction request; and transmitting a response to the interaction request to the client device, the response comprising further interaction information comprising the interaction information and further time information from the server according to the metadata structure, wherein one or more quality of experience (QoE) metrics are at least partially based on the further interaction information.

[0163]

[0168] In some embodiments of the additional embodiment method, at least one of the first time information and further time information may include one or more timestamps.

[0164]

[0169] Some embodiments of the additional embodiment method may further include receiving an identifier code and identifying the configuration of a metadata structure based on the received identifier code.

[0165]

[0170] Some embodiments of the additional embodiment method may further include receiving an identifier code and using the identifier code to check the processing order of user interactions on the server.

[0166]

[0171] Some embodiments of the additional embodiment method may further include receiving an identifier code and using the identifier code to identify a user interaction delay.

[0167]

[0172] In some embodiments of the additional embodiment method, identifying the user interaction delay may include receiving the time at which the user performed the interaction associated with the identifier code.

[0168]

[0173] Some embodiments of the additional embodiment method may further include identifying an interaction request drop rate, the interaction request drop rate may include the rate at which interaction requests are dropped by the server, and the interaction request drop rate may include a QoE metric.

[0169]

[0174] In some embodiments of the methods of the additional examples, at least one of the one or more QoE metrics may be user interaction delay.

[0170]

[0175] For some embodiments of the additional examples of the method, the response may include an updated description of the MR scene.

[0171]

[0176] For some embodiments of the methods of the additional embodiments, the response may include a new description of the MR scene.

[0172]

[0177] In some embodiments of the additional embodiment method, the response may include information indicating that the interaction request was dropped by the server.

[0173]

[0178] In some embodiments of the additional embodiment method, the response may include an encoded version of the rendered MR scene.

[0174]

[0179] In some embodiments of the additional examples of the method, further time information may also be included from the processing stage on the server.

[0175]

[0180] Some embodiments of the methods of the additional embodiments may further include receiving one or more quality-of-excitement (QoE) metrics that are at least partially based on further interaction information.

[0176]

[0181] Apparatus in additional embodiments according to several embodiments may include a processor and a non-temporary computer-readable medium that stores instructions that, when executed by the processor, are operable to cause the apparatus to perform any one of the claims listed above.

[0177]

[0182] It should be noted that one or more different hardware elements of the embodiments described herein are referred to as “modules” that perform (i.e., execute, perform, etc.) the various functions described herein in relation to each module. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that would be considered appropriate by those skilled in the art for a given implementation. It should be noted that each module described also includes executable instructions for performing one or more functions described as being performed by each module, and these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions and / or such, or include them, and may be stored in any suitable one or more non-temporary computer-readable media such as RAM, ROM, etc.

[0178]

[0183] While features and elements are described above in specific combinations, those skilled in the art will recognize that each feature or element can be used individually or in any combination with other features and elements. In addition, the methods described herein may be implemented in computer programs, software, or firmware embedded on a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital multi-purpose disks (DVDs). A processor may be used, together with software, to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.< / dot> < / dot> < / dot> < / dot> < / dot> < / dot> < / dot> < / dot> < / dot>

Claims

1. It is a method, The client device communicates with the server to configure a metadata structure for managing the processing of received interactions relating to a mixed reality (MR) scene, wherein the metadata structure includes a series of time information identifications performed by the client device and the server device when processing the received interactions. The system receives a user action related to the MR scene and, in accordance with the interaction information and metadata structure corresponding to the user action, starts processing the interaction request based on the user action. In the client device, first time information is identified in the first stage of processing the interaction request, In the first step, the interaction information is updated with the first time information, Sending the interaction request to the server along with the updated interaction information, Receiving a response from the server to the interaction request, wherein the response includes further interaction information, including the updated interaction information, and further time information from the server in accordance with the metadata structure. A method comprising identifying one or more quality-of-experience (QoE) metrics based at least partially on the updated further interaction information.

2. In the client device, the second time information is identified in the second stage of processing the interaction request, The method according to claim 1, further comprising updating the interaction information with the second time information in the second step.

3. The method according to claim 2, wherein at least one of the first time information and the second time information includes one or more timestamps.

4. In the client device, the second time information is identified in the second stage of processing the response from the server, In the second stage, the further interaction information is updated with the second time information based on the response from the server, The method according to any one of claims 1 to 3, further comprising:

5. Receiving an identifier code, Based on the received identifier code, the configuration of the metadata structure is identified, The method according to any one of claims 1 to 4, further comprising:

6. Receiving an identifier code, Using the aforementioned identifier code, the processing order of user interactions on the server is checked, The method according to any one of claims 1 to 5, further comprising:

7. Receiving an identifier code, The round-trip interaction delay is identified using the aforementioned identifier code, The method according to any one of claims 1 to 6, further comprising:

8. The method according to claim 7, wherein identifying the round-trip interaction delay includes storing the time the user performed the interaction associated with the identifier code.

9. This further includes identifying the interaction request drop rate, The interaction drop rate includes the rate at which interaction requests are dropped by the server. The method according to any one of claims 1 to 8, wherein the interaction request drop rate includes a QoE metric.

10. The method according to any one of claims 1 to 9, wherein at least one of the one or more QoE metrics is a round-trip interaction delay.

11. The method according to any one of claims 1 to 10, wherein the response includes an updated description of the MR scene.

12. The method according to any one of claims 1 to 11, wherein the response includes a new description of the MR scene.

13. The method according to any one of claims 1 to 12, wherein the response includes information indicating that the interaction request was dropped by the server.

14. The method according to any one of claims 1 to 13, wherein the response includes an encoded version of the rendered MR scene.

15. The method according to any one of claims 1 to 14, wherein the further time information further includes information from the processing stage in the server.

16. Processor and A non-temporary computer-readable medium that, when executed by the processor, stores instructions that cause the device to perform the method according to any one of claims 1 to 15, A device equipped with the following features.

17. It is a method, The server receives communications from a client device and configures a metadata structure for managing the processing of user interactions related to a mixed reality (MR) scene, wherein the metadata structure includes a series of time information identifications performed by the client device and the server device when processing the received interactions. The server receives an interaction request containing interaction information, Includes, The interaction information includes first time information in the first stage of processing the interaction request in the client device. In the server, further time information is identified in the second stage of processing the interaction request. Sending a response to the interaction request to the client device, wherein the response includes further interaction information including the interaction information, and further time information from the server according to the metadata structure, Includes, A method comprising one or more perceived quality (QoE) metrics, at least partially based on the aforementioned further interaction information.

18. The method according to claim 17, wherein at least one of the first time information and the further time information includes one or more timestamps.

19. Receiving an identifier code, Based on the received identifier code, the configuration of the metadata structure is identified, The method according to any one of claims 17 to 18, further comprising:

20. Receiving an identifier code, Using the aforementioned identifier code, the processing order of user interactions on the server is checked, The method according to any one of claims 17 to 19, further comprising:

21. Receiving an identifier code, The user interaction delay is identified using the aforementioned identifier code, The method according to any one of claims 17 to 20, further comprising:

22. The method according to claim 21, wherein determining the user interaction delay includes receiving the time at which the user performed the interaction associated with the identifier code.

23. This further includes identifying the interaction request drop rate, The interaction drop rate includes the rate at which interaction requests are dropped by the server. The aforementioned interaction request drop rate includes the QoE metric, The method according to any one of claims 17 to 22.

24. The method according to any one of claims 17 to 23, wherein at least one of the one or more QoE metrics is user interaction delay.

25. The method according to any one of claims 17 to 24, wherein the response includes an updated description of the MR scene.

26. The method according to any one of claims 17 to 25, wherein the response includes a new description of the MR scene.

27. The method according to any one of claims 17 to 26, wherein the response includes information indicating that the interaction request was dropped by the server.

28. The method according to any one of claims 17 to 27, wherein the response includes an encoded version of the rendered MR scene.

29. The method according to any one of claims 17 to 28, wherein the further time information further includes information from the processing stage in the server.

30. The method according to any one of claims 17 to 29, further comprising receiving one or more quality-of-experience (QoE) metrics based at least in part on the further interaction information.

31. Processor and A device comprising: a non-temporary computer-readable medium that, when executed by the processor, stores instructions that cause the device to perform the method according to any one of claims 17 to 30.

Images