Rendering policy based on confidence level of user data

Abstract

Systems and methods for rendering XR content based on confidence levels of tracked XR data. A method according to some embodiments includes obtaining at least a first tracked parameter (e.g., XR data, such as pose, gesture or gaze information, and other possibilities) and a first confidence level associated with the first tracked parameter; determining whether the first confidence level is below a first threshold associated with the first tracked parameter; and for at least one current frame of the augmented reality presentation, in response to determining that the first confidence level is below the first threshold, performing a predetermined first rendering action associated with the first tracked parameter. Possible predetermined actions are specified in a set of policies, defined for each XR data type and rendering type, and sent to the rendering engine when the XR application is established.

Description

[0001] Cross-references This application claims priority to European Patent Application No. 23306732.1, filed on October 9, 2023, entitled “Rendering strategy based on confidence level of user data”, which is incorporated herein by reference in its entirety. Background Technology

[0002] The 3GPP specification defines several architectures for Extended Reality (XR) applications / services. It specifies different modules that provide the XR experience to the user through their User Equipment (UE). The XR runtime module provides a set of functions for other modules, such as composition, peripheral device management, tracking, and spatial positioning. The rendering engine renders and provides different types of frames. One type of frame is the visual frame to be displayed on the UE display (visual renderer). The second type of frame is the audio frame to be sent to (multiple) UE speakers (audio renderers). The third type of frame is the haptic frame fed back to (multiple) UE actuators (haptic renderers). Additional modules include a scene manager and media access functionality.

[0003] The XR runtime and rendering engine can be on the same device (e.g., ... Figure 1 (Independent devices in the system), or they can be on two remote devices (e.g., on...) Figure 2 (in cloud-based or edge / segmentation rendering architectures).

[0004] In XR applications, the quality of visual and audio rendering of XR scenes depends on XR data, which can be described as information from the user, user experience (UE), and user environment. This information can include user / UE pose (view pose), gaze direction, trackable object pose, user input, hand / body pose, camera information, etc. This XR data can be generated from a set of sensors and provided by the XR runtime module to the rendering engine that provides visual, auditory, or haptic frames.

[0005] Open XR (an application programming interface (API) for XR applications) provides a set of functions or extensions to access different types of XR data. Among other things, these functions include the following: Use xrLocateViews() to retrieve the viewer's pose.

[0006] Use xrLocateSpace() to retrieve the pose of a trackable object.

[0007] • XR_EXT_hand_tracking and XR_FB_body_tracking extensions for detecting hand and body position and pose.

[0008] XR_FB_eye_tracking_social is used to obtain the position and orientation of the user's eyes.

[0009] The 3GPP SR_MSE TS 26.565 specification outlines a typical call flow in XR applications, and... Figure 3 The call flow can be described by referring to two main phases. The first phase is the establishment of the XR session: segmented rendering and connection establishment, as well as XR runtime configuration. During this phase, the UE and the segmented rendering server (SRS) exchange information about the XR space, user actions, view configuration, and projection format. The second phase is the rendering loop, scheduled at a predefined frame rate, which includes transferring XR data (pose information and user actions in the case of SR_MSE) from the XR runtime to the SRS, and transferring rendered frames from the SRS to the XR runtime. Summary of the Invention

[0010] The method according to some embodiments includes: obtaining at least a first tracked parameter (e.g., XR data, such as pose, posture, or gaze information, and other possibilities) and a first confidence level associated with the first tracked parameter; determining whether the first confidence level is lower than a first threshold associated with the first tracked parameter; and, for at least one current frame of extended reality rendering, performing a predetermined first rendering action associated with the first tracked parameter in response to determining that the first confidence level is lower than the first threshold.

[0011] In some embodiments, the predetermined first rendering action is at least one of the following: rendering the current frame using the most recent value of the tracked parameter associated with a confidence level not lower than a first threshold; rendering the most recently rendered frame instead of the current frame; rendering a still image instead of the current frame; or rendering a fade-out effect. Other actions may be used as alternatives or supplements.

[0012] In some embodiments, the first tracked parameter is at least one of the following: user pose; user gaze direction; pose of the trackable object; description of user input; or description of user pose.

[0013] In some embodiments, the method is executed by a client device in a standalone rendering architecture. In other embodiments, the method is executed by a segmented rendering server in a segmented rendering architecture.

[0014] When this method is executed by a segmented rendering server, in some embodiments, the method further includes receiving data from a client device in the segmented rendering architecture that is associated with a first tracked parameter, a first threshold, and a first rendering action. This data may be received during the establishment process of an extended reality session. In some embodiments, such data may be received in a data object, such as one or more policy objects having the syntax provided herein.

[0015] Some embodiments also include receiving multiple data objects during the setup process, each data object associating a corresponding tracked parameter with at least one corresponding threshold and at least one corresponding rendering action. In some embodiments, multiple thresholds are provided in an array, and multiple rendering actions are identified in a corresponding array, wherein each threshold corresponds to a specific one of the rendering actions.

[0016] In some embodiments, multiple data objects include corresponding identifiers to distinguish different data objects associated with the same tracked parameter.

[0017] In some embodiments, multiple data objects include corresponding priority indicators, and the determination of rendering actions is based at least in part on the priority indicators.

[0018] In some embodiments, multiple data objects include corresponding rendering type indicators, wherein the rendering type indicators indicate at least one of audio rendering, visual rendering, or haptic rendering. The specified rendering action may only affect the identified rendering type.

[0019] An apparatus according to some embodiments includes one or more processors configured to at least perform: obtaining at least a first tracked parameter and a first confidence level associated with the first tracked parameter; determining whether the first confidence level is lower than a first threshold associated with the first tracked parameter; for at least one current frame of extended reality rendering, rendering the current frame based on the first tracked parameter, at least in part based on the determination that the first confidence level is not lower than the first threshold; and for the current frame, performing a predetermined first rendering action associated with the first tracked parameter in response to determining that the first confidence level is lower than the first threshold. Attached Figure Description

[0020] Figure 1 The XR standalone architecture is shown.

[0021] Figure 2 The XR segmented rendering architecture is shown.

[0022] Figure 3 The traditional high-level call flow for segmenting presentation session establishment and operations is shown.

[0023] Figure 4The example embodiment illustrates a call flow managed using a presentation strategy based on the confidence level of the tracked data.

[0024] Figure 5 This is a flowchart of the rendering method executed in some embodiments.

[0025] Figure 6 This is a functional block diagram of an apparatus that can be used to implement a client and / or server in some embodiments. Detailed Implementation

[0026] for Figure 3 In cloud- or edge-based rendering, predictions (e.g., pose prediction) are used to compensate for the round-trip time required to render the virtual scene via a rendering loop. These predicted XR data are calculated by the UE (e.g., using extrapolated pose values) and sent to a segmented rendering server (SRS). They can also be calculated by the SRS based on raw XR data sent by the UE (sensor outputs and / or flags from the XR runtime module). These predicted XR data may be affected by low levels of accuracy.

[0027] like Figure 3 As shown, at 301, the segment rendering client creates a segment rendering session with the segment rendering server. At 302, a description of the segment rendering output is sent to the scene manager of the segment rendering client. At 303, a transport connection, such as a WebRTC session, is established between the scene manager and the segment rendering server. During the rendering loop, at 304, pose information and user actions from the XR runtime are received (e.g., via the XR source management module) and provided to the segment rendering server at 305. At 306, the segment rendering server performs rendering for the requested pose. At 307, the segment rendering server sends the next buffered frame to the media access function of the segment rendering client. At 308, the media access function decodes and processes the buffered frame. At 309, the media access function passes the raw buffered frame for display to the XR runtime, which then composites and renders the frame at 310.

[0028] However, even in standalone architectures where round-trip time is not an issue, sensor quality and lighting conditions can lead to inaccurate calculated / measured XR data (e.g., detection and tracking of trackable objects). This low accuracy of predicted or unpredicted XR data can be represented by a confidence level, calculated by the UE, and sent to the rendering engine along with the XR data.

[0029] For example, pose prediction confidence levels can affect rendering quality, and knowledge of these confidence levels can be used to improve Quality of Experience (QoE). Pose confidence levels can be estimated and sent to the rendering engine along with pose predictions. More generally, confidence levels can be estimated for each of many other XR data points that are fed into the rendering process and thus affect QoE.

[0030] The 3GPP SR_MSE TS 26.565 specification specifies a set of XR data (pose and user actions) sent to the rendering engine. The embodiments described herein relate to the determination and use of confidence level information associated with various types of this XR data. Other embodiments for resolving confidence levels for other types of XR data, not yet addressed in this invention, may be specified in future versions of this document. Example embodiments further describe methods implemented by the rendering engine to make appropriate decisions when the confidence level of one or more XR data sets is too low.

[0031] Example embodiments include actions that the rendering engine can use to render appropriate frames based on the confidence level of the XR data. In some embodiments, information characterizing these methods may be sent to the rendering engine during the establishment of the XR session. During the rendering loop, the rendering engine examines the XR data and its confidence level, and makes rendering decisions based on the appropriate actions.

[0032] Regarding the confidence level for pose prediction, a threshold can be defined for each type of XR data. Different types of decisions are triggered when the confidence level falls below the threshold. For visual rendering, in response to determining that the confidence level is below the threshold, one of the following actions can be taken: • Provides the final rendered image.

[0033] • Render the image based on the last correct XR data (with a confidence level above a threshold). This action can be used to account for some modifications to the 3D scene that may have occurred independently of the XR data (such as animated 3D objects, physical simulations, etc.).

[0034] • Provide users with still images to display, such as messages.

[0035] For audio rendering, in response to determining that the confidence level is below a threshold, one of the following actions can be taken: • Render audio frames based on the last correct XR data (with a confidence level above the threshold).

[0036] • Initiate the rendering of a series of audio frames until the confidence value reaches the correct value, for example, to create a fade-out effect before muting the sound, or to create a non-spatial version of the original audio content.

[0037] For haptic rendering, in response to determining that the confidence level is below a threshold, one of the following actions can be taken: • Render haptic frames based on the last correct XR data (with a confidence level above the threshold).

[0038] • Initiate a series of haptic frames to render until the confidence value returns to the correct value, for example, to create a fade-out effect before stopping the haptic effect, or to create a non-directional version of the original haptic content.

[0039] In the example embodiment, the rendering engine is configured with information indicating which action to perform based on the confidence level of the XR data. In a standalone architecture, the actions(s) can be set by the AR / MR application during scene loading. In a cloud- or edge-based architecture, the actions(s) can be negotiated during session establishment between the UE and the server, such as during segmented rendering session establishment. The information indicating the actions(s) to be performed can be included in the XR runtime configuration data transmitted during segmented rendering session creation.

[0040] Figure 4 A flowchart illustrating the rendering method that can be executed in some embodiments is shown. In step 401, the creation of a segmented rendering session includes providing XR configuration information, which includes information identifying actions (or "policies") to be implemented in the event of a low confidence level in the XR data. At 402, a description of the segmented rendering output is sent to the scene manager of the segmented rendering client. At 403, a transport connection, such as a WebRTC session, is established between the scene manager and the segmented rendering server. At 404 and 405, pose information, user actions, and other XR data are transmitted, including information about the confidence level of the data. In some embodiments, a unique ID may be provided to the XR data to distinguish several XR data of the same type, such as multiple trackable objects in a scene, multiple user poses, etc. At 406, the segmented rendering server performs rendering for the requested pose based on a rendering policy regarding the XR data and its confidence level. At 407, the segmented rendering server sends the next (multiple) buffered frames to the media access function of the segmented rendering client. At 408, the media access function decodes and processes the buffered frames. At 409, the media access function passes the raw buffered frame for display to the XR runtime, which then composes and renders the frame at 410.

[0041] The example implementation defines a method executed by the renderer when the confidence level of the XR data is below a threshold. If the confidence level is above the threshold, the XR data is considered good enough for computation of a new frame.

[0042] Below are examples of actions that can be performed in response to determining that one or more parameters of XR data have a confidence level below a threshold. The following list uses examples related to audio, visual, and haptic rendering, but the listed actions can also be used for other types of rendering.

[0043] • USE_LAST_DATA (for audio, visual, and haptic rendering): The last value of the XR data with the correct confidence level is used to render new frames. The correct XR data is saved by the rendering engine for later use.

[0044] • USE_LAST_FRAME (for visual rendering): Uses the last correctly rendered visual frame, even if some other XR data has the correct confidence level. One frame is rendered at a time from a set of correct XR data, and this frame is stored by the rendering engine for later use.

[0045] • USE_STILL_FRAME (for visual rendering): Provides a still image, which may optionally display a message to the user. In some embodiments, the still image may be accompanied by a silent audio frame. In some cases, prompts (via audio and / or video) may be presented on how to improve erroneous XR data. For example, prompts may provide suggestions on moving more slowly or increasing or decreasing lighting intensity. In some cases, a waiting message may be displayed to the user.

[0046] • USE_FADING_FRAME (for visual, audio, and haptic rendering): Initiates a series of fade-out frames until the confidence level returns to the correct value. This series can, for example, begin with a frame associated with the last correct XR data. When the confidence level returns to the correct value, a new frame is rendered using the correct XR data.

[0047] In some cases, an operation can be implemented globally on all XR data. In other cases, one or more actions can be applied to each type of XR data.

[0048] In some embodiments, the renderer is provided as an array of "policy" objects to configure it to handle data at low confidence levels. In one example, each policy object contains the renderer type, the type of XR data, the data ID, the threshold, and information indicating the action to be performed. Table 1 shows the syntax of an example policy object. Data types use enumerate Optional enumerate Optional number Optional floating point Forced enumerate Forced

[0049] Table 1 In Table 1 and other grammar examples described herein, the indication that a grammar element is "mandatory" or "optional" only indicates whether a grammar rule is mandatory or optional according to that particular embodiment. A "mandatory" grammar element in one embodiment may be optional or even omitted entirely in different embodiments.

[0050] The syntax elements of the strategy object in Table 1 can convey the following information.

[0051] render_type: The type of renderer that the strategy object targets. · ALL (All) · VISUAL (Visual) • AUDIO · HAPTIC (tactile sensation) · [other].

[0052] If this parameter is set to "All" or if it does not exist, the strategy object is applied to all renderers.

[0053] data_type: The type of XR data for which the operation is performed. For example, it might have the following possible values: · ALL (All) • VIEW_POSE (view pose) used for user / UE pose. • GAZE_POSE (gaze posture) used for gaze direction. • TRACKABLE POSE (trackable object pose) for the pose of a trackable object. • USER_ACTION (user action) used for user input. • GESTURE_POSE (pose) is used to describe the user's hand / body posture. • Any other value that identifies the additional XR data.

[0054] If this parameter is set to ALL, or if this parameter does not exist, an action that can be applied to all XR data types will be selected.

[0055] data_id: A unique ID for XR data. Multiple elements in an XR scene can generate XR data of the same type (scene). (Multiple trackable objects, multiple user gestures, etc.). Furthermore, this XR data may have different characteristics defined by different policy objects. Same action. If this parameter is not present, the policy object will be applied to all objects with the same data_type. This id corresponds to... The ID that is sent along with each XR data during the rendering cycle at XR runtime.

[0056] Threshold: A confidence level value; levels below this value (or equal to or lower than this value) are considered invalid. Confidence levels can be... It is a floating-point number between 0 and 1. In some embodiments, it can have a finite number of possible values ​​(e.g., 0, 1 / 8, 2 / ...). 8……,7 / 8、1)。

[0057] Decision: The action to be applied that may have the following values ​​as described above: • USE_LAST_DATA (use the last data) • USE_LAST_FRAME (use the last frame) • USE_STILL_FRAME (Use still frames) • Use_FADING_FRAME (use fade-in / fade-out frames) · [other].

[0058] In some embodiments, several strategy objects can be specified for a given XR data. This can be done, for example, by defining different thresholds corresponding to different rendering decisions. For instance, the xr_strategies array can include a strategy object using USE_STILL_FRAME for confidence levels below 2 / 8, and another strategy object using USE_LAST_DATA for confidence levels below 5 / 8 (i.e., between 2 / 8 and 5 / 8). If the confidence level is above 5 / 8, the XR data is considered valid for processing new rendered images.

[0059] Below is an example of an array of strategy objects using JSON syntax, although different syntax may be used in different implementations.

[0060] .

[0061] In some embodiments, one or more actions are described in an array of policy objects (e.g., only one object per XR data type), including the type of XR data, a list of thresholds, and a list of corresponding actions to be applied. This improves the implementation of several policies for an XR data. Data types use enumerate Optional enumerate Optional number Optional floating-point array Forced Enumeration array Forced

[0062] Table 2 The syntax elements of the strategy object in Table 2 can convey the following information.

[0063] render_type, data_type, and data_id: Same as the parameters mentioned above.

[0064] thresholds: An array of thresholds considered for the data_type mentioned above, as described above.

[0065] Decision: An array of values ​​indicating the actions to be performed, where each action corresponds to a threshold value at the same index in the threshold array. value.

[0066] The following is an example of a strategy object description using the syntax elements of Table 2, implemented using JSON syntax: .

[0067] In the example embodiment, when the rendering engine receives the set of XR data and its confidence level, it determines whether the confidence level is below a threshold, and in response to the determination that the confidence level is below the threshold, the rendering engine renders the frame according to the indicated action.

[0068] There may be situations where inconsistent actions are triggered based on low confidence levels (e.g., USE_LAST_DATA and USE_LAST_FRAME). In some embodiments, this can be resolved by implementing a default action, such as the USE_LAST_FRAME action. In some embodiments, different actions can be assigned different priorities by default or through configuration, and this can be resolved by implementing a triggering action with higher priority. In some embodiments, priority can be represented by an additional parameter (e.g., priority) in the policy object, which is set to a value between 0 (low priority) and 5 (high priority). The absence of this parameter in the policy object may indicate low priority. Below is an example of a policy object description implementing the priority parameter using JSON syntax.

[0069] .

[0070] Figure 5 This is a flowchart of a method performed in some embodiments. In some embodiments, Figure 5 The method is performed by the segmented rendering server. In some embodiments, this is where the method is more effective. Figure 5 The method can be understood as occurring after the establishment process, during which a policy object (or other similar data) has already been received from the client. In other embodiments, Figure 5 The method is executed by the client itself.

[0071] like Figure 5 As shown, during runtime, information about different types of tracked XR data and their confidence levels is obtained at steps 502 and 504. For those types of tracked data associated with predetermined thresholds, the confidence levels are compared to the corresponding thresholds to identify any type of tracked data with confidence levels below the thresholds at step 506. If none are found, XR frame rendering can proceed as normal at step 509. At step 511, the rendered frames can be stored for later use should one of the confidence levels in a future frame drops below its corresponding threshold.

[0072] If at 506 it is determined that one or more confidence levels have fallen below their corresponding thresholds, then at 508 rendering actions associated with those thresholds are identified. If none of the identified rendering actions are incompatible with each other, then all such actions can be implemented. If some actions are incompatible (e.g., displaying a new frame relative to a previously tracked data display), then it can be determined which rendering action to perform. As shown at 510, this determination can be made based on priority data (e.g., from a policy object), or otherwise, for example, actions associated with lower confidence levels can be given priority over actions associated with higher confidence levels. Based on this determination, one or more selected rendering actions are performed. The actions performed can be, for example, implementing a fade-in / fade-out effect (512), rendering a new frame using previously tracked data (514), displaying a still image such as a notification or recommendation to the user (516), or displaying a previously rendered frame (518).

[0073] Example System The encoding, decoding, processing, and rendering of XR content described here can use methods such as Figure 6 This is achieved through systems like [system name missing]. Figure 6 This is a block diagram illustrating examples of systems in which various aspects and embodiments are implemented. System 1000 can be implemented as a device including the various components described below and configured to perform one or more aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described herein.

[0074] System 1000 includes at least one processor 1010 configured to execute instructions loaded thereon for implementing various aspects, such as those described in this document. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0075] System 1000 includes an encoder / decoder module 1030, which is configured to, for example, process data to provide an encoding or decoding scenario, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents multiple modules that can be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both encoding and decoding modules. Furthermore, the encoder / decoder module 1030 may be implemented as a separate element of system 1000, or it may be integrated within processor 1010 as a combination of hardware and software known to those skilled in the art.

[0076] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described in this document may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more of various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input scenes, decoding scenes or portions of decoding scenes, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0077] In some embodiments, the memory within the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be the processor 1010 or encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamically volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (General Video Coding, a new standard developed by JVET, the Joint Video Experts Group).

[0078] As shown in box 1130, inputs can be provided to the components of system 1000 through various input devices. Such input devices include, but are not limited to: (i) an RF section that receives radio frequency (RF) signals, for example, transmitted over the air by a broadcasting device; (ii) component (COMP) input terminals (or a set of COMP input terminals); (iii) universal serial bus (USB) input terminals; and / or (iv) high-definition multimedia interface (HDMI) input terminals.

[0079] In various embodiments, as known in the art, the input device of block 1130 has associated respective input processing elements. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting the signal band to a band), (ii) down-converting the selected signal, (iii) further band-limiting to a narrower band to select, for example, a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements to perform these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0080] Furthermore, the USB and / or HDMI terminals may include their respective interface processors for connecting the system 1000 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) can be implemented as needed, for example, within a separate input processing IC or within the processor 1010. Similarly, aspects of USB or HDMI interface processing can be implemented as needed, either within a separate interface IC or within the processor 1010. Demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0081] Various components of system 1000 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data between them using a suitable connection arrangement 1140 (e.g., internal buses as known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0082] System 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0083] In various embodiments, data is streamed to or otherwise provided to system 1000 using a wireless network such as a Wi-Fi network (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)). In these embodiments, the Wi-Fi signal is received via a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that delivers data via an HDMI connection of input block 1130 to provide streaming data to system 1000. Still other embodiments use an RF connection of input block 1130 to provide streaming data to system 1000. As indicated above, various embodiments provide data in a non-streaming manner. Furthermore, various embodiments use wireless networks instead of Wi-Fi, such as cellular networks or Bluetooth networks.

[0084] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a flexible display, and / or a foldable display. Display 1100 can be used in televisions, tablets, laptops, cellular phones (mobile phones), or other devices. Display 1100 can also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop). In various examples of embodiments, other peripheral devices 1120 include one or more of a standalone digital video disc (or digital multifunction disc) (DVD for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functionality based on the output of system 1000. For example, a disc player performs the function of playing the output of system 1000.

[0085] In various embodiments, signaling such as AV links, Consumer Electronics Control (CEC), or other communication protocols enabling inter-device control with or without user intervention is used to communicate control signals between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120. Output devices may be communicatively coupled to system 1000 via dedicated connections through their respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speaker 1110 may be integrated into a single unit with other components of system 1000 in electronic devices such as, for example, a television set. In various embodiments, display interface 1070 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0086] For example, if the RF portion of input 1130 is part of a separate set-top box, then display 1100 and speaker 1110 can alternatively be separate from one or more of the other components. In various embodiments where display 1100 and speaker 1110 are external components, the output signal can be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0087] System 1000 may include one or more sensor devices 1095. Examples of sensor devices that may be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as user position and orientation. Where system 1000 is used as a control module (such as control modules 124, 1254) for an augmented reality display, the user's position and orientation can be used to determine how image data is rendered so that the user perceives the correct portion of a virtual object or scene from the correct perspective. In the case of a head-mounted display device, the device's own position and orientation can be used to determine the user's position and orientation for the purpose of rendering virtual content. In the case of other display devices such as telephones, tablets, computer monitors, or televisions, other inputs can be used to determine the user's position and orientation for the purpose of rendering content. For example, the user can use a touchscreen, keypad or keyboard, trackball, joystick, or other inputs to select and / or adjust the desired viewpoint and / or direction of observation. When the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation used for rendering content can be selected and / or adjusted based on the movement of the display device.

[0088] The embodiments can be implemented by computer software implemented by processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, memory 1020 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 1010 can be of any type suitable for the technical environment and can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0089] Further embodiments A method according to some embodiments includes: obtaining a plurality of tracked parameters and associated confidence levels; identifying at least a first tracked parameter among the tracked parameters having an associated first confidence level below a first threshold, the first threshold being associated with the first tracked parameter; identifying a first rendering action associated with the lower-threshold confidence level of the first tracked parameter; and performing the first rendering action.

[0090] Some embodiments further include: identifying at least a second tracked parameter among the tracked parameters, having an associated second confidence level below a second threshold, the second threshold being associated with the second tracked parameter; identifying a second rendering action associated with the lower-threshold confidence level of the second tracked parameter; determining whether the second rendering action is compatible with the first rendering action; and performing the second rendering action in response to determining that the second rendering action is compatible with the first rendering action.

[0091] Some embodiments further include: identifying at least a second tracked parameter among the tracked parameters, having an associated second confidence level below a second threshold, the second threshold being associated with the second tracked parameter; identifying a second rendering action associated with the lower-threshold confidence level of the second tracked parameter; and determining whether the second rendering action is compatible with the first rendering action; wherein, in response to determining that the second rendering action is incompatible with the first rendering action, the second rendering action is not performed.

[0092] Some embodiments further include: identifying at least a second tracked parameter among the tracked parameters, having an associated second confidence level below a second threshold, the second threshold being associated with the second tracked parameter; identifying a second rendering action associated with the lower-threshold confidence level of the second tracked parameter; and determining whether the second rendering action has a higher priority than the first rendering action; wherein, in response to determining that the second rendering action does not have a higher priority than the first rendering action, the second rendering action is not executed.

[0093] The method according to some embodiments includes: obtaining at least a first tracked parameter (e.g., XR data, such as pose, posture, or gaze information, and other possibilities) and a first confidence level associated with the first tracked parameter; determining whether the first confidence level is lower than a first threshold, the first threshold being associated with the first tracked parameter; and for at least one current frame of extended reality rendering, in response to determining that the first confidence level is lower than the first threshold, performing a predetermined first rendering action associated with the first tracked parameter.

[0094] In some embodiments, the predetermined first rendering action is at least one of the following: rendering the current frame using the most recent value of the tracked parameter associated with a confidence level not lower than a first threshold; rendering the most recently rendered frame instead of the current frame; rendering a still image instead of the current frame; or rendering a fade-out effect. Other actions may be used as alternatives or supplements.

[0095] In some embodiments, the first tracked parameter is at least one of the following: user pose; user gaze direction; pose of the trackable object; description of user input; or description of user pose.

[0096] In some embodiments, the method is executed by a client device in a standalone rendering architecture. In other embodiments, the method is executed by a segmented rendering server in a segmented rendering architecture.

[0097] When this method is executed by a segmented rendering server, in some embodiments, the method further includes receiving data from a client device in the segmented rendering architecture that is associated with a first tracked parameter, a first threshold, and a first rendering action. This data may be received during the establishment process of an extended reality session. In some embodiments, such data may be received in a data object, such as one or more policy objects having the syntax provided herein.

[0098] Some embodiments also include receiving multiple data objects during the setup process, each data object associating a corresponding tracked parameter with at least one corresponding threshold and at least one corresponding rendering action. In some embodiments, multiple thresholds are provided in an array, and multiple rendering actions are identified in a corresponding array, wherein each threshold corresponds to a specific one of the rendering actions.

[0099] In some embodiments, multiple data objects include corresponding identifiers to distinguish different data objects associated with the same tracked parameter.

[0100] In some embodiments, multiple data objects include corresponding priority indicators, and the determination of rendering actions is based at least in part on the priority indicators.

[0101] In some embodiments, multiple data objects include corresponding rendering type indicators, wherein the rendering type indicators indicate at least one of audio rendering, visual rendering, or haptic rendering. The specified rendering action may only affect the identified rendering type.

[0102] An apparatus according to some embodiments includes one or more processors configured to at least perform: obtaining at least a first tracked parameter and a first confidence level associated with the first tracked parameter; determining whether the first confidence level is lower than a first threshold, the first threshold being associated with the first tracked parameter; rendering, for at least one current frame of extended reality presentation, based on the first tracked parameter, at least in part based on determining that the first confidence level is not lower than the first threshold; and for the current frame, performing a predetermined first rendering action associated with the first tracked parameter in response to determining that the first confidence level is lower than the first threshold.

[0103] This disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described and are generally described in a manner that may sound restrictive, at least for the purpose of illustrating individual characteristics. However, this is for the purpose of clarity and does not limit the disclosure or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, the aspects described can also be combined and interchanged with aspects described in earlier applications.

[0104] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically shown, other embodiments are also contemplated, and the discussion of particular embodiments does not limit the breadth of implementations. At least one aspect generally relates to encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions thereon stored thereon for encoding or decoding XR content data according to any of the described methods, and / or computer-readable storage media having a bitstream generated according to any of the described methods stored thereon.

[0105] This document describes various methods, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first" and "second" can be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of these terms does not imply an ordering of the modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or in a time period overlapping with the second decoding.

[0106] For example, various numerical values ​​may be used in this disclosure. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0107] The embodiments described herein can be implemented by computer software, or by a combination of hardware and software, using a processor or other hardware. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be of any type suitable for the technical environment and can include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0108] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0109] The implementations and aspects described herein can be implemented, for example, in a method or process, apparatus, software program, data stream, or signal. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the discussed features can also be implemented in other forms (e.g., apparatus or program). An apparatus can be implemented, for example, in appropriate hardware, software, and firmware. A method can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. A processor also includes communication devices, such as, for example, a computer, a cellular phone, a portable / personal digital assistant (“PDA”), and other devices that facilitate information communication between end users.

[0110] References to “an embodiment” or “an embodiment” or “an implementation” or “implementation”, and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the appearance of the phrase “in an embodiment” or “in an embodiment” or “in an implementation” or “in an implementation”, and any other variations appearing throughout the various places of this disclosure, do not necessarily refer to the same embodiment.

[0111] Additionally, this disclosure may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimated information, calculated information, predicted information, or information retrieved from memory.

[0112] Furthermore, this disclosure can refer to "accessing" various information fragments. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0113] Additionally, this disclosure may refer to "receiving" various pieces of information. Like "access," receiving is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) one or more. Furthermore, "receiving" is generally referred to in one way or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0114] It will be understood that the use of any of the following " / ", "and / or", and "...at least one", such as in the cases of "A / B", "A and / or B", and "at least one of A and B", is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to as many items as listed.

[0115] Furthermore, as used herein, the word “signal” specifically refers, among other things, to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals a specific parameter among a plurality of parameters used for region-based filter parameter selection for artifact removal filtering. In this way, in embodiments, the same parameter can be used on both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicitly signal) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without transmission (implicitly signal) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It will be appreciated that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the verb form of the word “signal” has been referred to above, the word “signal” can be used (e.g., also) as a noun herein.

[0116] The implementation can generate various signals, which are formatted to carry information, such as information that can be stored or transmitted. This information may include, for example, instructions for executing a method, or data generated by one of the implementations. For example, the signal may be formatted to carry a bitstream of the embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal can be transmitted via various wired or wireless links. The signal may be stored on a processor-readable medium.

[0117] We have described numerous embodiments. Features of these embodiments may be provided individually or in any combination across various claim classes and types. Furthermore, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination, across various claim classes and types: • A bitstream or signal including one or more of the syntax elements or variations thereof.

[0118] • Includes bit streams or signals that convey the syntax of information generated according to any of the described embodiments.

[0119] • Create and / or transmit and / or receive and / or decode bit streams or signals including one or more of the syntax elements or their variants.

[0120] • Creation and / or transmission and / or reception and / or decoding according to any of the described embodiments.

[0121] • Methods, processes, apparatus, media for storing instructions, media for storing data, or signals according to any of the described embodiments.

[0122] Note that the various hardware elements in one or more of the described embodiments may be referred to as “modules,” which implement (i.e., carry out, execute, etc.) the various functions described herein in conjunction with the respective modules. As used herein, a module includes hardware considered suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices). Each described module may also include executable instructions for carrying out one or more functions described as being performed by the respective module, and it should be noted that these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., or include hardware instructions, firmware instructions, software instructions, etc., and may be stored in any suitable one or more non-transitory computer-readable media (such as commonly referred to as RAM, ROM, etc.).

[0123] Although the features and elements have been described above in specific combinations, each feature or element can be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware contained in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROMs and digital multifunction discs (DVDs). The processor associated with the software can be used to implement a radio frequency transceiver used in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: Obtain multiple tracked parameters and their associated confidence levels; At least a first tracked parameter is identified among the tracked parameters, which has an associated first confidence level below a first threshold, the first threshold being associated with the first tracked parameter; Identify the first rendering action associated with a confidence level below a threshold for the first tracked parameter; as well as Perform the first rendering action.

2. An apparatus comprising one or more processors configured to perform at least: Obtain multiple tracked parameters and their associated confidence levels; At least a first tracked parameter is identified among the tracked parameters, which has an associated first confidence level below a first threshold, the first threshold being associated with the first tracked parameter; Identify the first rendering action associated with a confidence level below a threshold for the first tracked parameter; as well as Perform the first rendering action.

3. The method according to claim 1 or the apparatus according to claim 2, further comprising: At least a second tracked parameter is identified among the tracked parameters, which has an associated second confidence level below a second threshold, the second threshold being associated with the second tracked parameter; Identify a second rendering action that is associated with a confidence level below a threshold for the second tracked parameter; Determine if the second rendering action is compatible with the first rendering action; as well as In response to determining that the second rendering action is compatible with the first rendering action, the second rendering action is executed.

4. The method according to claim 1 or the apparatus according to claim 2, further comprising: At least a second tracked parameter is identified among the tracked parameters, which has an associated second confidence level below a second threshold, the second threshold being associated with the second tracked parameter; Identify a second rendering action that is associated with a confidence level below a threshold for the second tracked parameter; as well as Determine if the second rendering action is compatible with the first rendering action; Specifically, in response to determining that the second rendering action is incompatible with the first rendering action, the second rendering action is not executed.

5. The method according to claim 1 or the apparatus according to claim 2, further comprising: At least a second tracked parameter is identified among the tracked parameters, which has an associated second confidence level below a second threshold, the second threshold being associated with the second tracked parameter; Identify a second rendering action that is associated with a confidence level below a threshold for the second tracked parameter; as well as Determine whether the second rendering action has a higher priority than the first rendering action; Specifically, in response to determining that the second rendering action does not have a higher priority than the first rendering action, the second rendering action is not executed.

6. The method according to claim 1 or claims 3-5 which are dependent on claim 1, or the apparatus according to claim 2 or claims 3-5 which are dependent on claim 2, wherein the first rendering action is at least one of the following: The current frame is rendered using the most recent value of the tracked parameter associated with a confidence level not lower than a first threshold. The most recently rendered frame is used instead of the current frame; Present a still image instead of the current frame; or It creates a fade-out effect.

7. The method according to claim 1 or claims 3-5 which are dependent on claim 1, or the apparatus according to claim 2 or claims 3-5 which are dependent on claim 2, wherein the first rendering action and the second rendering action are two different actions selected from the following actions: The current frame is rendered using the most recent value of the tracked parameter associated with a confidence level not lower than a first threshold. The most recently rendered frame is used instead of the current frame; Present a still image instead of the current frame; or It creates a fade-out effect.

8. The method according to claim 1 or claims 3-7 which are dependent on claim 1, or the apparatus according to claim 2 or claims 3-7 which are dependent on claim 2, wherein the plurality of tracked parameters includes at least two different parameters selected from the following: User attitude; User's gaze direction; The orientation of the trackable object; The description entered by the user; or Description of user gestures.

9. The method according to claim 1 or claims 3-5 which are dependent on claim 1, or the apparatus according to claim 2 or claims 3-5 which are dependent on claim 2, wherein the first tracked parameter and the second tracked parameter are two different parameters selected from the following parameters: User attitude; User's gaze direction; The orientation of the trackable object; The description entered by the user; or Description of user gestures.

10. The method according to claim 1 or claims 3-9 which are dependent on claim 1, or the apparatus according to claim 2 or claims 3-9 which are dependent on claim 2, implemented by a client device in a standalone rendering architecture.

11. The method according to claim 1 or the apparatus according to claims 3-9 of claim 1, or the apparatus according to claim 2 or the apparatus according to claims 3-9 of claim 2, implemented by a segmented rendering server in a segmented rendering architecture.

12. The method according to claim 11 of claim 1, or the apparatus according to claim 11 of claim 2, further comprising receiving data associated with the first tracked parameter, the first threshold, and the first rendering action from a client device in a segmented rendering architecture.

13. The method according to claim 12, which is dependent on claim 1, or the apparatus according to claim 12, which is dependent on claim 2, wherein data associated with the first tracked parameter, the first threshold, and the first rendering action are received during the establishment process of the extended reality session.

14. The method according to claim 13 of claim 1, or the apparatus according to claim 13 of claim 2, further comprising receiving a plurality of data objects during the setup process, each data object associating a corresponding tracked parameter with at least one corresponding threshold and at least one corresponding rendering action.

15. The method according to claim 14, which is dependent on claim 1, or the apparatus according to claim 14, which is dependent on claim 2, wherein the plurality of data objects include corresponding priority indicators, and wherein the execution of the first rendering action is at least partially conditional on the priority indicators.

Images