Distributed attitude prediction

Abstract

Systems and techniques for generating and / or processing virtual content in a distributed system (e.g., a distributed augmented reality system) are described herein. For example, a first device (e.g., a client device) of a distributed augmented reality system may send one or more sets of attitude prediction parameters (e.g., prediction coefficients, predicted times associated with raw attitude data, and / or raw attitude data) to a second device (e.g., a server device) of the distributed augmented reality system. The second device may predict one or more poses of the second device based on the set of pose prediction parameters, and may generate virtual content based on a pose of the predicted poses of the first device. The second device may send the virtual content, and the first device may receive the virtual content. The first device may then display one or more virtual objects based at least in part on the received virtual content.

Description

Technical Field

[0001] The present disclosure generally relates to attitude estimation techniques. For example, aspects of the present disclosure include systems and techniques for providing distributed attitude prediction. Background Art

[0002] Pose estimation can be used in various applications such as extended reality (XR) (e.g., virtual reality, augmented reality, mixed reality), computer vision, and robotics to determine the position and orientation of a human or object relative to a scene or environment. Pose information can be used to manage the interaction between a human or object and a particular scene or environment. For example, the pose of a user or a device worn by a user can be used to enhance or augment the user's real or physical environment with virtual content. As another example, the pose (e.g., position and orientation) of a robot can be used to allow the robot to manipulate objects or avoid collisions with objects when moving around in a scene.

[0003] As noted above, pose estimation can be used in XR systems such as smart glasses and head-mounted displays (HMDs). Such extended reality systems generally implement cameras and sensors to track the positioning of XR devices and other objects within the physical environment. XR devices can use this tracking information to provide realistic XR experiences to users of XR devices. For example, XR devices can allow users to experience or interact with immersive virtual environments or content. In order to provide a realistic XR experience, XR technology can integrate virtual content with the physical world or a video feed depicting a scene in the physical world. In some cases, XR technology can match the relative pose and movement of objects and devices in the physical world. For example, in addition to the pose of features of devices, objects, and / or real-world environments, XR devices can also use tracking information to calculate the relative pose of the XR system or device to match the relative positioning and movement of devices, objects, and / or real-world environments. In some cases, XR devices can use the pose and movement of one or more devices, objects, and / or real-world environments to render content relative to the real-world environment in a convincing manner. Relative pose information can be used to match virtual content to the user's perceived motion and the spatiotemporal state of the device, objects, and real-world environment.

[0004] However, the computational complexity of the pose estimation system can impose significant power and resource requirements and can be a limiting factor in various applications. The computational complexity of pose estimation can also limit the performance and scalability of tracking and localization applications that rely on pose information. Summary of the invention

[0005] Systems and techniques for providing distributed (e.g., server-based) gesture prediction (e.g., in an extended reality (XR) system) are described. According to at least one example, a method for generating virtual content at a first device in a distributed extended reality system is provided. The method includes: receiving one or more gesture prediction parameter sets from a second device associated with the distributed extended reality system, wherein each corresponding gesture prediction parameter set from the one or more gesture prediction parameter sets includes a corresponding prediction coefficient and a corresponding prediction time associated with corresponding original gesture data; predicting one or more gestures of the second device based on the one or more gesture prediction parameter sets; generating virtual content based on a gesture in the predicted one or more gestures of the first device; and sending the virtual content to the second device.

[0006] In another example, a first device for generating virtual content in a distributed extended reality system is provided, the first device comprising: at least one memory and at least one processor coupled to the at least one memory (e.g., configured in a circuit system). The at least one processor is configured to: receive one or more gesture prediction parameter sets from a second device associated with the distributed extended reality system, wherein each corresponding gesture prediction parameter set from the one or more gesture prediction parameter sets includes a corresponding prediction coefficient and a corresponding prediction time associated with corresponding original gesture data; predict one or more gestures of the second device based on the one or more gesture prediction parameter sets; generate virtual content based on a gesture in the predicted one or more gestures of the first device; and send the virtual content to the second device.

[0007] In another example, a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: receive one or more posture prediction parameter sets from a second device associated with the distributed extended reality system, wherein each corresponding posture prediction parameter set from the one or more posture prediction parameter sets includes a corresponding prediction coefficient and a corresponding prediction time associated with corresponding original posture data; predict one or more postures of the second device based on the one or more posture prediction parameter sets; generate virtual content based on a posture among the predicted one or more postures of the first device; and send the virtual content to the second device.

[0008] In another example, a first device for generating virtual content in a distributed extended reality system is provided. The device includes: a component for receiving one or more gesture prediction parameter sets from a second device associated with the distributed extended reality system, wherein each corresponding gesture prediction parameter set from the one or more gesture prediction parameter sets includes a corresponding prediction coefficient and a corresponding prediction time associated with corresponding original gesture data; a component for predicting one or more gestures of the second device based on the one or more gesture prediction parameter sets; a component for generating virtual content based on a gesture in the predicted one or more gestures of the first device; and a component for sending the virtual content to the second device.

[0009] According to at least one additional example, a method for processing virtual content at a first device in a distributed extended reality system is provided. The method includes: sending one or more gesture prediction parameter sets to a second device associated with the distributed extended reality system, wherein each corresponding gesture prediction parameter set from the one or more gesture prediction parameter sets includes a corresponding prediction coefficient and a corresponding prediction time associated with corresponding original gesture data; receiving virtual content based on a gesture in one or more predicted gestures of the first device from the second device, the one or more predicted gestures being predicted based on the one or more gesture prediction parameter sets sent to the second device; and displaying one or more virtual objects based at least in part on the received virtual content.

[0010] In another example, a first device for processing virtual content in a distributed extended reality system is provided, the first device comprising: at least one memory and at least one processor coupled to the at least one memory (e.g., configured in a circuit system). The at least one processor is configured to: send one or more gesture prediction parameter sets to a second device associated with the distributed extended reality system, wherein each corresponding gesture prediction parameter set from the one or more gesture prediction parameter sets includes a corresponding prediction coefficient and a corresponding prediction time associated with corresponding original gesture data; receive virtual content based on a gesture in one or more predicted gestures of the first device from the second device, the one or more predicted gestures being predicted based on the one or more gesture prediction parameter sets sent to the second device; and display one or more virtual objects based at least in part on the received virtual content.

[0011] In another example, a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: send one or more gesture prediction parameter sets to a second device associated with the distributed extended reality system, wherein each corresponding gesture prediction parameter set from the one or more gesture prediction parameter sets includes a corresponding prediction coefficient and a corresponding prediction time associated with corresponding original gesture data; receive virtual content based on a gesture in one or more predicted gestures of the first device from the second device, the one or more predicted gestures being predicted based on the one or more gesture prediction parameter sets sent to the second device; and display one or more virtual objects based at least in part on the received virtual content.

[0012] In another example, a first device for processing virtual content in a distributed extended reality system is provided. The device includes: a component for sending one or more gesture prediction parameter sets to a second device associated with the distributed extended reality system, wherein each corresponding gesture prediction parameter set from the one or more gesture prediction parameter sets includes a corresponding prediction coefficient and a corresponding prediction time associated with corresponding original gesture data; a component for receiving virtual content based on a gesture in one or more predicted gestures of the first device from the second device, the one or more predicted gestures being predicted based on the one or more gesture prediction parameter sets sent to the second device; and a component for displaying one or more virtual objects based at least in part on the received virtual content.

[0013] In some aspects, one or more of the devices described herein are, are part of, and / or include an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a mobile device (e.g., a mobile phone or other mobile device), a wearable device, a head mounted device (HMD) device, a wireless communication device, a camera, a personal computer, a laptop computer, a server computer or a server device (e.g., an edge or cloud-based server, a personal computer acting as a server device, a mobile device such as a mobile phone acting as a server device, an XR device acting as a server device, a network router, or other device acting as a server), a vehicle or a computing device component of a vehicle, another device, or a combination thereof. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the devices described above may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0014] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0015] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Illustrative examples of the present application are described in detail below with reference to the following drawings:

[0017] Figure 1 is a diagram illustrating an example of a distributed extended reality (XR) system according to aspects of the present disclosure;

[0018] Figure 2 is a diagram illustrating an example of a device according to aspects of the present disclosure;

[0019] Figure 3 is a diagram illustrating another example of a distributed XR system according to aspects of the present disclosure;

[0020] Figure 4 is a diagram illustrating an example of an XR system according to aspects of the present disclosure;

[0021] Figure 5is a diagram illustrating an example configuration of a client device according to aspects of the present disclosure;

[0022] Fig. 6A is a diagram illustrating an example of a server device configured to predict or extrapolate a gesture at fixed time intervals according to aspects of the present disclosure;

[0023] Figure 6B is a diagram illustrating an example of a gesture extrapolation engine according to aspects of the present disclosure;

[0024] Fig. 7A is a diagram illustrating an example of a server device configured to predict or extrapolate a gesture when determining that virtual content is to be generated according to aspects of the present disclosure;

[0025] Figure 7B is a diagram illustrating another example of a gesture extrapolation engine according to aspects of the present disclosure;

[0026] Figure 8 is a diagram illustrating an example of a deep neural network supporting server-side posture prediction according to aspects of the present disclosure;

[0027] Fig. 9 is a flow chart illustrating an example of a process for supporting server-side gesture prediction according to aspects of the present disclosure;

[0028] Fig.10 is a flowchart illustrating an example of a process of generating virtual content at a server device of a distributed extended reality system according to aspects of the present disclosure;

[0029] Fig.11 is a flow chart illustrating an example of a process for processing virtual data at a client device of a distributed extended reality system according to aspects of the present disclosure; and

[0030] Fig.12 is a diagram illustrating an example of a computing system according to aspects of the present disclosure. DETAILED DESCRIPTION

[0031] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is apparent that various aspects can be implemented without these specific details. Each drawing and description are not intended to be restrictive.

[0032] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0033] An extended reality (XR) system or device may provide virtual content to a user and / or may combine the real world and physical environment with a virtual environment (composed of virtual content) to provide an XR experience to the user. The real world environment may include real world objects (also referred to as physical objects), such as people, vehicles, buildings, tables, chairs, and / or other real world objects or physical objects. An XR system or device may facilitate interaction with different types of XR environments (e.g., a user may use an XR system or device to interact with an XR environment). An XR system may include a VR system that facilitates interaction with a virtual reality (VR) environment, an AR system that facilitates interaction with an augmented reality (AR) environment, an MR system that facilitates interaction with a mixed reality (MR) environment, and / or other XR systems. As used herein, the terms XR system and XR device are used interchangeably. Examples of XR systems or devices include head-mounted displays (HMDs), smart glasses, and the like. In some cases, an XR system may track a user's location (e.g., a user's hands and / or fingertips) to allow the user to interact with items of virtual content.

[0034] AR is a technology that provides virtual or computer-generated content (referred to as AR content) on a user's view of a physical, real-world scene or environment. AR content may include any virtual content, such as video, images, graphical content, location data (e.g., global positioning system (GPS) data or other location data), sound, any combination thereof, and / or other enhanced content. AR systems are designed to enhance (or enhance) rather than replace people's current perception of reality. For example, a user can see a real static or moving physical object through an AR device display, but the user's visual perception of the physical object can be enhanced or enhanced by a virtual image of the object (e.g., a real-world car replaced by a virtual image of a DeLorean), by AR content added to a physical object (e.g., virtual wings added to a live animal), by AR content displayed relative to a physical object (e.g., information virtual content displayed near a sign on a building, a virtual coffee cup virtually anchored to a real-world table in one or more images (e.g., placed on top of the real-world table), etc.) and / or by displaying other types of AR content. Various types of AR systems can be used for games, entertainment, and / or other applications.

[0035] VR provides a complete immersive experience in a three-dimensional computer-generated VR environment or video that depicts a virtual version of a real-world environment. The VR environment can interact in a seemingly real or physical way. As the user experiencing the VR environment moves in the real world, the image rendered in the virtual environment also changes, giving the user the perception that the user is moving in the VR environment. For example, the user can turn left or right, look up or down, and / or move forward or backward, thereby changing the user's viewpoint of the VR environment. The VR content presented to the user can change accordingly, making the user's experience as seamless as in the real world. VR content may include VR video in some cases, which can be captured and rendered at very high quality, thereby potentially providing a real immersive virtual reality experience. Virtual reality applications may include games, training, education, sports videos, online shopping, and the like. VR content can be rendered and displayed using a VR system or device such as a VR HMD or other VR headset that completely covers the user's eyes during the VR experience.

[0036] MR technology can combine aspects of VR and AR to provide an immersive experience for users. For example, in an MR environment, real-world and computer-generated objects can interact (e.g., real people can interact with virtual people as if the virtual people were real people).

[0037] In some cases, an XR system or device may match the relative pose and movement of objects and devices in the physical world. For example, an XR device may use tracking information to calculate the relative pose of features of a device, an object, and / or a real-world environment in order to match the relative positioning and movement of the device, an object, and / or a real-world environment. In some cases, an XR device may use the pose and movement of one or more devices, objects, and / or a real-world environment to render content relative to the real-world environment in a convincing manner. Relative pose information may be used to match virtual content to the user's perceived motion and the spatiotemporal state of the device, objects, and the real-world environment.

[0038] As mentioned previously, the computational complexity of the pose estimation system may impose significant power and resource requirements and may be a limiting factor in various applications. The computational complexity of pose estimation may also limit the performance and scalability of tracking and positioning applications that rely on pose information. For example, when implementing XR applications, the computational complexity of pose estimation may impose large power and resource requirements on the device. Such power and resource requirements are exacerbated by the recent trend to implement such technologies in mobile and wearable devices and to make these devices smaller, lighter, and more comfortable (e.g., by reducing the heat emitted by the device) to be worn by users for longer periods of time.

[0039] In some cases, a distributed XR system (which may also be referred to herein as a split XR system) may be used to reduce the computational complexity of a client device by dividing or splitting the XR processing burden between a first device or system (e.g., including a client device such as an XR headset, HMD, etc.) and a second device or system (e.g., including one or more server devices). In a distributed XR system, a first device (e.g., a client device) may send posture information (e.g., HMD posture data) representing the posture of the client device to a second device (e.g., a server device). The posture information may include orientation information (e.g., pitch, roll, and yaw) and / or translation information (e.g., horizontal displacement, vertical displacement, and depth displacement). The second device (e.g., a server device) may render virtual content (e.g., XR content) at a specific rendering call based on the posture information of the first device (e.g., the client device). The second device (e.g., the server device) may then compress the virtual content and send the virtual content to the first device (e.g., the client device) along with the rendering posture information defining the posture (referred to as the rendering posture) in which the rendered content is located. The first device (eg, the client device) may decompress and render the virtual content after distorting the virtual content using the rendering pose and the latest pose information of the first device (eg, the client device).

[0040] The data path of sending gesture information from a first device (e.g., a client device) to a second device (e.g., a server device) and receiving virtual content and rendered gestures from the second device at the first device is called a round trip. There is a round trip delay in the round trip, which refers to the delay between when the gesture is determined by the first device and when the rendered virtual content is received at the first device. Due to the round trip delay, the first device can predict future gestures by approximating the round trip time. By predicting future gestures, the first device can ensure that the difference or increment between the rendered gesture and the displayed gesture is small. If the increment is large, there may be distortion artifacts that occur in the displayed virtual content.

[0041] Network interference may occur when sending gesture information from a first device (e.g., a client device) to a second device (e.g., a server device), which may be due to overlapping channels in different wireless networks. For example, if the signal strength of the second network is low in an area of ​​the environment where the distributed XR system is located, the distributed XR system in the first network may experience low interference from the second network, while if the signal strength of the third network is high in the area of ​​the environment, the distributed XR system may experience severe interference from the third network. When severe interference is present, gesture information may not be received on the second device at a fixed pace or time interval (e.g., gesture information is received sporadically across time).

[0042] If the gesture does not arrive before the render call in which the second device (e.g., the server device) renders the virtual content, the second device may need to select an old or outdated gesture to render the virtual content. When the first device (e.g., the client device) displays the virtual content generated by the server device using the outdated gesture, the virtual content is visually perceived as a translational jitter that degrades the user experience. In some cases, approximately 5% to 15% of the rendered virtual content frames are generated using outdated gestures under severe interference channel conditions.

[0043] Systems, apparatuses, electronic devices, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") for providing distributed pose prediction (e.g., in extended reality (XR) systems) are described herein. In some aspects, a first device (e.g., a client device, such as an XR device including an HMD or other type of XR device) may send pose prediction parameters to a second device (e.g., a device acting as a server device, such as a cloud- or edge-based server, a personal computer acting as a server, another XR device acting as a server, etc.). Although the first device will be referred to herein as a client device and the second device will be described herein as a server device, the systems and techniques described herein may be implemented by any device in a distributed system (e.g., a distributed XR system).

[0044] The posture prediction parameter set may include raw posture data, prediction coefficients associated with the raw posture data, and a prediction time associated with the time when the raw posture data was obtained. The raw posture data may be obtained by the client device using one or more sensors (e.g., samples from an inertial measurement unit (IMU)) or a tracking system (e.g., a 6-degree-of-freedom (6DOF) tracker) that obtains sensor data from the one or more sensors. In some cases, the raw posture data includes pitch, roll, yaw, and translation data (e.g., horizontal displacement, vertical displacement, and depth displacement relative to a reference point) representing the posture of the client device. In some examples, coefficients may be used in a polynomial to predict the posture of the client device. For example, the coefficients may be bias-corrected IMU coefficients.

[0045] In some aspects, a server device may use a gesture prediction parameter set (e.g., original gesture data, prediction coefficients, and prediction time) to predict or extrapolate gestures at fixed time intervals or pace. In such aspects, even if the arrival of gesture parameters at the server device is intermittent, the rendering engine of the rendering virtual content on the server device will continue to observe evenly spaced gesture samples. In some cases, the server device may use a gesture prediction parameter set (e.g., original gesture, prediction coefficients, and prediction time) to predict or extrapolate gestures when determining that virtual content is to be generated (e.g., in response to receiving a gesture request from the rendering engine, such as when rendering a call). In such aspects, a server device may use a gesture prediction parameter set (e.g., original gesture data, prediction coefficients, and prediction time) to generate gestures just before a rendering call. For example, a rendering engine may request a gesture from a gesture extrapolation engine. Based on the most recent prediction parameter set (e.g., original gesture data, prediction coefficients, and prediction time), a gesture extrapolation engine may predict or extrapolate gestures and output the predicted gestures to the rendering engine.

[0046] The client device has access to finely sampled sensor (e.g., IMU) values, so the client can generate accurate coefficients that are important for high-quality posture prediction. By transmitting the coefficients generated by the client to the server device, the system can ensure high accuracy of posture extrapolation. Such systems and techniques for predicting or extrapolating postures at the server device are more accurate than predicting postures based on actual predicted postures received on the server device to extrapolate postures. By predicting or extrapolating postures based on posture prediction data (e.g., original posture data, prediction coefficients, and prediction time) on the server device, visual artifacts (e.g., translation jitter) will be eliminated or reduced and the experience of the user of the client device will be improved based on using less or no outdated postures to render frames. In addition, a reduction in end-to-end latency can be achieved because the posture can be extrapolated just before the rendering call. The posture can also be sent at a lower rate, which reduces competition for channel access.

[0047] In some aspects, a server device and a client device (e.g., in a distributed XR system) may access a shared clock. For example, in a phone-to-XR device (also known as phone-to-glasses) distributed system in which the server device is a phone and is wirelessly connected to the XR device, the underlying communication layer (e.g., a WiFi layer) may provide clock functionality to both devices to which it is providing wireless connections. In such an example, one or more updates to the predicted time at the server end may be performed based on the common clock (e.g., in the event that the coefficients do not arrive in time). In one example, if the server device is configured to generate gesture samples every 2 milliseconds (ms), then in the absence of a common clock, the server device will generate gestures every 2ms in the server clock. In the case of a common clock, the server device may generate gesture samples every 2ms on the client device clock, if necessary. This example can be used to solve synchronization issues between gesture generation events on the server side and prediction coefficient generation events on the client side.

[0048] In some cases, in a distributed XR system, rendered frames may arrive irregularly from a server device at a client device, such as depending on rendering complexity and network jitter. However, the display of the client device may operate or run at a fixed frame rate. Therefore, in some aspects (e.g., where a common clock is available between the server device and the client device), the server device may more accurately predict at which vsync opportunity (display tick) the rendered frame will be displayed and the future time point at which the clock tick will occur when rendering a virtual frame. Based on this determination, the server device may determine a prediction time and use the determined prediction time in conjunction with a pose prediction coefficient to obtain a predicted pose.

[0049] Various aspects of the application will be described with reference to the accompanying drawings.

[0050] Figure 11 is a diagram illustrating an example of a system 100 according to aspects of the present disclosure. System 100 may represent a distributed extended reality (XR) system. System 100 may include device 105, server 110 (also referred to as server device), and storage 115. Although system 100 illustrates two devices 105, a single server 110, a single storage 115, and a single network 120, the present disclosure is applicable to any system architecture having one or more devices 105, servers 110, storage 115, and networks 120. In some cases, storage 115 may be part of server 110. Device 105, server 110, and storage 115 may communicate with each other via network 120 using communication link 125 and exchange information supporting server-based pose prediction for XR, such as multimedia packets, multimedia data, multimedia control information, pose prediction parameters (e.g., raw pose data, prediction coefficients, and corresponding prediction times). In some cases, portions of the techniques described herein for providing server-based pose prediction for XR may be performed by one or more of devices 105 and portions of the techniques may be performed by server 110 , or both.

[0051] Device 105 may be an XR device (e.g., a head mounted display (HMD), XR glasses such as virtual reality (VR) glasses, augmented reality (AR) glasses, etc.), a mobile device (e.g., a cellular phone, a smart phone, a personal digital assistant (PDA), etc.), a wireless communication device, a tablet computer, a laptop computer, and / or other devices that support various types of communication and functional features related to multimedia (e.g., sending, receiving, broadcasting, streaming, aggregating, capturing, storing, and recording multimedia data). UE 105 may additionally or alternatively be referred to by those skilled in the art as user equipment (UE), user device, smart phone, Bluetooth device, Wi-Fi device, mobile station, subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, and / or some other suitable terminology. In some cases, device 105 may also be capable of communicating directly with another device (e.g., using a peer-to-peer (P2P) or device-to-device (D2D) protocol, such as using sidelink communication). For example, device 105 may be capable of receiving or sending various information, such as instructions or commands (e.g., multimedia related information), from another device 105 or to the other device.

[0052] The device 105 may include an application 130 and a multimedia manager 135. Although the system 100 illustrates that the device 105 includes both the application 130 and the multimedia manager 135, the application 130 and the multimedia manager 135 may be optional features of the device 105. In some cases, the application 130 may be a multimedia-based application that may receive (e.g., download, stream, broadcast) or send (e.g., upload) multimedia data to the server 110, the storage device 115, or another device 105 via the use of the communication link 125.

[0053] The multimedia manager 135 may be part of a general purpose processor, a digital signal processor (DSP), an image signal processor (ISP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), discrete gate or transistor logic components, discrete hardware components, or any combination thereof, or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described in the present disclosure. For example, the multimedia manager 135 may process multimedia (e.g., image data, video data, audio data) from a local memory or storage device 115 of the device 105 and / or write multimedia data to the local memory or the storage device.

[0054] The multimedia manager 135 may also be configured to provide multimedia enhancement, multimedia restoration, multimedia analysis, multimedia compression, multimedia streaming, and multimedia synthesis, among other functionality. For example, the multimedia manager 135 may perform white balancing, cropping, scaling (e.g., multimedia compression), adjusting resolution, multimedia stitching, color processing, multimedia filtering, spatial multimedia filtering, artifact removal, frame rate adjustment, multimedia encoding, multimedia decoding, and multimedia filtering. By further example, in accordance with the techniques described herein, the multimedia manager 135 may process multimedia data to support server-based pose prediction for XR.

[0055] The server 110 may be a data server, a cloud server, a server associated with a multimedia subscription provider, a proxy server, a web server, an application server, a communication server, a home server, a mobile server, or any combination thereof. In some cases, the server 110 may include a multimedia distribution platform 140. The multimedia distribution platform 140 may allow the device 105 to discover, browse, share, and download multimedia via the network 120 using the communication link 125, and thus provide digital distribution of multimedia from the multimedia distribution platform 140. In this way, digital distribution may be in the form of delivering media content (such as audio, video, images) through an online delivery medium (such as the Internet) without using physical media. For example, the device 105 may upload or download multimedia-related applications for streaming, downloading, uploading, processing, promoting, etc. multimedia (e.g., images, audio, video). The server 110 may also transmit a variety of information to the device 105, such as instructions or commands (e.g., multimedia-related information) to download multimedia-related applications on the device 105.

[0056] The storage device 115 may store a variety of information, such as instructions or commands (e.g., multimedia related information). For example, the storage device 115 may store multimedia 145. The device may support server-based posture prediction for XR associated with the multimedia 145. The device 105 may retrieve the stored data from the storage device 115 via the network 120 using the communication link 125. In some examples, the storage device 115 may be a memory device (e.g., read-only memory (ROM), random access memory (RAM), cache memory, buffer memory, etc.) that stores a variety of information such as instructions or commands (e.g., multimedia related information), a relational database (e.g., a relational database management system (RDBMS) or a structured query language (SQL) database), a non-relational database, a network database, an object-oriented database, or other types of databases. The storage device 115 may store a reference model (e.g., a machine learning model, a simulation model, etc.), and in some cases, the machine learning model may be updated based on user input received from the device 105.

[0057] The network 120 may provide encryption, access authorization, tracking, Internet Protocol (IP) connectivity, and other access, computation, modification, and / or functionality. Examples of the network 120 may include any combination of cloud networks, local area networks (LANs), wide area networks (WANs), virtual private networks (VPNs), wireless networks (e.g., using 802.11), cellular networks (using third generation (3G), fourth generation (4G), long term evolution (LTE), or new radio (NR) systems (e.g., fifth generation (5G))), etc.). The network 120 may include the Internet.

[0058] The communication link 125 shown in the system 100 may include uplink transmissions from the device 105 to the server 110 and the storage device 115, and / or downlink transmissions from the server 110 and the storage device 115 to the device 105. The wireless link 125 can transmit bidirectional communications and / or unidirectional communications. In some examples, the communication link 125 can be a wired connection or a wireless connection or both. For example, the communication link 125 may include one or more connections, including but not limited to Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), cellular, Z-WAVE, 802.11, peer-to-peer, LAN, wireless local area network (WLAN), Ethernet, FireWire, fiber optics, and / or other connection types associated with wireless communication systems.

[0059] In some examples, device 105 or server 110 may process information (e.g., from storage device 115) via machine learning techniques (e.g., using one or more machine learning models, such as one or more neural networks). For example, device 105 may be configured to use machine learning to determine one or more posture prediction parameters (e.g., prediction coefficients). In some cases, server 110 may process posture prediction parameters (e.g., raw posture data, prediction coefficients, and corresponding prediction times) received from client device 105 to predict the current and / or future posture of device 105 (e.g., corresponding to the head posture of the user of device 105). In some cases, data from storage device 115 may include information from other users (e.g., head posture patterns of other users experiencing similar XR applications). Examples of machine learning techniques may include linear regression, logistic regression, decision trees, support vector machines (SVM), naive Bayes, k-nearest neighbors, random forests, dimensionality reduction algorithms, gradient boosting algorithms, or any combination thereof.

[0060] Figure 2 2 is a diagram illustrating an example of a device 200. The device 200 is a client device (eg, Figure 1 device 105) or a server device (e.g., Figure 1 10). As shown, the device 200 includes a central processing unit (CPU) 210 with a CPU memory 215, a GPU 225 with a GPU memory 230, a display 245, a display buffer 235 storing data associated with rendering, a user interface unit 205, and a system memory 240. For example, the system memory 240 may store a GPU driver 220 (illustrated as being included in the CPU 210 described below) with a compiler, a GPU program, a natively compiled GPU program, etc. The user interface unit 205, the CPU 210, the GPU 225, the system memory 240, the display 245, and the extended reality manager 250 can communicate with each other (e.g., using a system bus).

[0061] Examples of CPU 210 include, but are not limited to, a digital signal processor (DSP), a general-purpose microprocessor, an application-specific integrated circuit (ASIC), a field programmable logic array (PGA), or other equivalent integrated or discrete logic circuits. Figure 2 245. The CPU 210 and the GPU 225 may be integrated into a single unit, although they are illustrated as separate units in the example of FIG. 245, in some examples, the CPU 210 and the GPU 225 may be integrated into a single unit. The CPU 210 may execute one or more software applications. Examples of applications may include an operating system, a word processor, a web browser, an email application, a spreadsheet, a video game, an audio and / or video capture, playback or editing application, or other such applications that initiate the generation of image data to be presented via the display 245. As illustrated, the CPU 210 may include a CPU memory 215. For example, the CPU memory 215 may represent an on-chip storage device or memory used when executing machine or object code. The CPU memory 215 may include one or more volatile or non-volatile memory or storage devices, such as flash memory, magnetic data media, optical storage media, etc. The CPU 210 may be able to read or write values ​​from or to the CPU memory 215 faster than reading or writing values ​​from or to the system memory 240, which may be accessed, for example, via a system bus.

[0062] GPU 225 can represent one or more special processors for performing graphics operations. That is, for example, GPU 225 can be a special hardware unit with fixed functions and programmable components for rendering graphics and executing GPU applications. GPU 225 can also include DSP, general-purpose microprocessor, ASIC, FPGA or other equivalent integrated or discrete logic circuits. GPU 225 can be constructed with a highly parallel structure, which provides more efficient processing of complex graphics-related operations than CPU 210. For example, GPU 225 may include multiple processing elements configured to operate multiple vertices or pixels in a parallel manner. The highly parallel nature of GPU 225 can allow GPU 225 to generate graphics images (e.g., graphical user interface and two-dimensional or three-dimensional graphics scenes) for display 245 faster than CPU 210.

[0063] In some instances, GPU 225 may be integrated into the motherboard of device 200. In other instances, GPU 225 may be present on a graphics card installed in a port in the motherboard of device 200, or may be otherwise incorporated into a peripheral device configured to interoperate with device 200. As illustrated, GPU 225 may include GPU memory 230. For example, GPU memory 230 may represent an on-chip storage device or memory used when executing machine or object code. GPU memory 230 may include one or more volatile or non-volatile memory or storage devices, such as flash memory, magnetic data media, optical storage media, etc. GPU 225 may be able to read or write values ​​from GPU memory 230 faster than reading or writing values ​​from system memory 240, which may be accessed, for example, via a system bus. That is, GPU 225 may read data from and write data to GPU memory 230 without using the system bus to access off-chip memory. This operation may allow GPU 225 to operate in a more efficient manner by reducing the need for GPU 225 to read and write data via a system bus, which may be experiencing heavy bus traffic.

[0064] Display 245 represents a unit capable of displaying video, images, text, or any other type of data for consumption by a viewer. Display 245 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED), an active matrix OLED (AMOLED), etc. Display buffer 235 represents a memory or storage device dedicated to storing data for image presentation, such as computer-generated graphics, static images, video frames, etc. for display 245. Display buffer 235 may represent a two-dimensional buffer including a plurality of storage locations. In some cases, the number of storage locations within display buffer 235 may roughly correspond to the number of pixels to be displayed on display 245. For example, if display 245 is configured to include 640×480 pixels, display buffer 235 may include 640×480 storage locations storing pixel color and intensity information (such as red, green, and blue pixel values ​​or other color values). Display buffer 235 may store the final pixel value of each of the pixels processed by GPU 225. Display 245 may retrieve the final pixel values ​​from display buffer 235 and display the final image based on the pixel values ​​stored in display buffer 235 .

[0065] The user interface unit 205 represents a unit with which a user can interact or otherwise interface to communicate with other units of the device 200, such as the CPU 210. Examples of the user interface unit 205 include, but are not limited to, a trackball, a mouse, a keyboard, and other types of input devices. The user interface unit 205 may also be or include a touch screen, and the touch screen may be incorporated as part of the display 245.

[0066] System memory 240 may include one or more computer-readable storage media. Examples of system memory 240 include, but are not limited to, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPM), compact disk read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer or processor. System memory 240 can store program modules and / or instructions that can be accessed by CPU 210 for execution. Additionally, system memory 240 can store user applications and application surface data associated with the applications. In some cases, system memory 240 can store information for use by other components of device 200 and / or information generated by these other components. For example, system memory 240 can act as device memory for GPU 225, and can store data to be operated by GPU 225 and data from operations performed by GPU 225.

[0067] In some examples, system memory 240 may include instructions that cause CPU 210 or GPU 225 to perform functions attributed to CPU 210 or GPU 225 in various aspects of the present disclosure. In some examples, system memory 240 may be considered a non-transitory storage medium. The term "non-transitory" should not be interpreted to mean that system memory 240 is non-removable. As an example, system memory 240 may be removed from device 200 and moved to another device. As another example, a system memory substantially similar to system memory 240 may be inserted into device 200. In some examples, a non-transitory storage medium may store data that changes over time (e.g., in RAM).

[0068] System memory 240 may store GPU driver 220 and compilers, GPU programs, and natively compiled GPU programs. GPU driver 220 may represent a computer program or executable code that provides an interface to access GPU 225. CPU 210 may execute GPU driver 220 or a portion thereof to interface with GPU 225, and for this reason, GPU driver 220 may be executed in the CPU 210. Figure 22 is shown as being within CPU 210 in the example of FIG. 2. GPU driver 220 may be accessed by programs executed by CPU 210 or other executable programs, including GPU programs stored in system memory 240. Thus, when one of the software applications executing on CPU 210 requires graphics processing, CPU 210 may provide graphics commands and graphics data to GPU 225 for rendering to display 245 (e.g., via GPU driver 220).

[0069] In some cases, the GPU program may include code written in a high-level (HL) programming language (e.g., using an application programming interface (API)). Examples of APIs include Open Graphics Library ("OpenGL"), DirectX, Render-Man, WebGL, or any other public or proprietary standard graphics API. Instructions may also conform to so-called heterogeneous computing libraries, such as Open Computing Language ("OpenCL"), DirectCompute, etc. In general, an API includes a predetermined set of standardized commands executed by associated hardware. API commands allow a user to instruct the hardware components of GPU 225 to execute commands without the user knowing the specifics of the hardware components. In order to process graphics rendering instructions, CPU 210 may (e.g., through GPU driver 220) issue one or more rendering commands to GPU 225 to cause GPU 225 to perform some or all of the rendering of graphics data. In some examples, the graphics data to be rendered may include a list of graphics primitives (e.g., points, lines, triangles, quadrilaterals, etc.).

[0070] The GPU program stored in the system memory 240 may call or otherwise include one or more functions provided by the GPU driver 220. The CPU 210 generally executes a program in which the GPU program is embedded, and when encountering the GPU program, the GPU program is passed to the GPU driver 220. The CPU 210 executes the GPU driver 220 in this context to process the GPU program. That is, for example, the GPU driver 220 may process the GPU program by compiling the GPU program into an object or machine code that can be executed by the GPU 225. This object code may be referred to as a natively compiled GPU program. In some examples, a compiler associated with the GPU driver 220 may operate in real time or near real time to compile the GPU program during the execution of the program in which the GPU program is embedded. For example, a compiler generally represents a unit that reduces the HL instructions defined according to the HL programming language to the low-level (LL) instructions of the LL programming language. After compilation, these LL instructions can be executed by a specific type of processor or other type of hardware such as FPGA, ASIC, etc. (including but not limited to CPU 210 and GPU 225).

[0071] exist Figure 2 In an example of , the compiler may receive the GPU program from CPU 210 when executing the HL code including the GPU program. That is, the software application being executed by CPU 210 may call GPU driver 220 (e.g., via a graphics API) to issue one or more commands to GPU 225 for rendering one or more graphics primitives into a displayable graphics image. The compiler may compile the GPU program to generate a natively compiled GPU program that conforms to the LL programming language. The compiler may then output the natively compiled GPU program including LL instructions. In some examples, the LL instructions may be provided to GPU 225 in the form of a list of drawing primitives (e.g., triangles, rectangles, etc.).

[0072] LL instructions (e.g., which may alternatively be referred to as primitive definitions) may include vertex specifications that specify one or more vertices associated with a primitive to be rendered. The vertex specifications may include positioning coordinates for each vertex, and in some instances, other attributes associated with the vertex, such as color coordinates, normal vectors, and texture coordinates. Primitive definitions may include primitive type information, scaling information, rotation information, and the like. Based on instructions issued by a software application (e.g., a program in which a GPU program is embedded), the GPU driver 220 may formulate one or more commands that specify one or more operations to be performed by the GPU 225 in order to render the primitive. When the GPU 225 receives the command from the CPU 210, the GPU may decode the command and configure one or more processing elements to perform the specified operation, and may output the rendered data to the display buffer 235.

[0073] The GPU 225 may receive a locally compiled GPU program, and then in some instances, the GPU 225 renders one or more images and outputs the rendered images to the display buffer 235. For example, the GPU 225 may generate a plurality of primitives to be displayed at the display 245. Primitives may include one or more of lines (including curves, splines, etc.), points, circles, ellipses, polygons (e.g., triangles), or any other two-dimensional primitives. The term "primitive" may also refer to three-dimensional primitives, such as cubes, cylinders, spheres, cones, pyramids, tori, etc. In general, the term "primitive" refers to any basic geometric shape or element that can be rendered by the GPU 225 to be displayed as an image (or a frame in the context of video data) via the display 245. The GPU 225 may transform primitives and other attributes of primitives (e.g., defining color, texture, lighting, camera configuration, or other aspects) into the so-called "world space" by applying one or more model transformations (which may also be specified in the state data). Once transformed, the GPU 225 may apply a view transform to the active camera (which may also be specified in the state data defining the camera) to transform the coordinates of the primitives and lights into camera or eye space. The GPU 225 may also perform vertex shading to render the appearance of the primitives according to any active lights. The GPU 225 may perform vertex shading in one or more of the above model, world, or view space.

[0074] Once the primitives are shaded, the GPU 225 may perform projection to project the image into the canonical view volume. After transforming the model from eye space to the canonical view volume, the GPU 225 may perform clipping to remove any primitives that do not reside entirely within the canonical view volume. For example, the GPU 225 may remove any primitives that are not within the camera's frame. The GPU 225 may then map the coordinates of the primitives from the view volume to the screen space, effectively reducing the three-dimensional coordinates of the primitives to two-dimensional coordinates of the screen. Given the transformed and projected vertices that define the primitives and their associated shading data, the GPU 225 may then rasterize the primitives. Generally speaking, rasterization may refer to the task of taking an image described in a vector graphics format and converting the image into a raster image (e.g., a pixelated image) for output on a video display or storage in a bitmap file format.

[0075] The GPU 225 may include a dedicated fast grid buffer (e.g., a fast memory buffer such as GMEM that can be referred to by the GPU memory 230). As discussed herein, the rendering surface may be divided into several grids. In some cases, the grid size is determined by the format (e.g., pixel color and depth information) and the rendering target resolution divided by the total amount of GMEM. The number of grids may vary based on the device 200 hardware, the target resolution size, and the target display format. The rendering pass may draw pixels (e.g., render, write, etc.) into the GMEM (e.g., using a high bandwidth that matches the capabilities of the GPU). The GPU 225 may then parse the GMEM (e.g., write the mixed pixel values ​​from the GMEM as a single-layer burst to the display buffer 235 or the frame buffer in the system memory 240). This may be referred to as grid-based or tile-based rendering. When all grids are completed, the driver may swap buffers and start the grid process again for the next frame.

[0076] For example, the GPU 225 may implement a tile-based architecture that renders an image or render target by splitting the image into multiple parts (referred to as tiles or bins). The bins may be sized based on the size of the GPU memory 230 (e.g., which may alternatively be referred to herein as GMEM or cache), the resolution of the display 245, the color or Z precision of the render target, etc. When implementing tile-based rendering, the GPU 225 may perform a bin pass and one or more rendering passes. For example, with respect to a bin pass, the GPU 225 may process the entire image and categorize the rasterized primitives into bins.

[0077] The device 200 may use sensor data, sensor statistics, or other data from one or more sensors. Some examples of monitored sensors may include an IMU, an eye tracker, a tremor sensor, a heart rate sensor, etc. In some cases, an IMU may be included in the device 200 and may use some combination of an accelerometer, a gyroscope, or a magnetometer to measure and report specific force, angular velocity, and sometimes the orientation of the body.

[0078] In some cases, device 200 may include an extended reality manager 250. The extended reality manager may implement deep learning-based techniques to determine one or more coefficients, which may be sent to a server device (e.g., server device 110) along with other pose prediction parameters (e.g., raw pose data and predicted times corresponding to the raw pose data). For example, extended reality manager 250 may utilize a deep learning system (e.g., one or more neural networks) to determine the one or more coefficients based on sensor data from one or more sensors (e.g., from one or more IMUs). Further details regarding the one or more coefficients are described herein. In some cases, extended reality manager 250 may implement aspects of extended reality, augmented reality, virtual reality, etc.

[0079] As described herein, the server device may use the pose prediction parameters to predict the pose of device 200. For example, the server device may identify and generate a virtual object based on the predicted pose according to the techniques described herein.

[0080] Figure 3 An example of an extended reality system 300 supporting server-side based gesture prediction for extended reality according to aspects of the present disclosure is illustrated. In some examples, the extended reality system 300 can implement aspects of the system 100 and the device 200. In some cases, the extended reality system 300 may include a device 305, a network 320, and a communication link 325, which may be as described in reference to FIG. Figure 1 305, communication link 125, and network 120. As described herein, device 305 (e.g., worn by user 310) can account for head motion 315 based on a deep learning model. In some cases, device 305 can be an extended reality device, which can generally implement aspects of extended reality, virtual reality, augmented reality, etc. A system including device 305, network 320, or other elements in extended reality system 300 can be referred to as an extended reality system.

[0081] Device 305 may overlay virtual objects with real-world objects in view 330. For example, view 330 may generally refer to visual input to user 310 via device 305, a display generated by device 305, a virtual object configuration generated by device 305, etc. For example, view 330-A may refer to visible real-world objects at some initial time and visible virtual objects overlaid on or co-existing with the real-world objects. View 330-B may refer to visible real-world objects at some later time and visible virtual objects overlaid on or co-existing with the real-world objects. As discussed herein, positioning differences in real-world objects (e.g., and thus overlaid virtual objects) may result from view 330-A shifting to view 330-B at 335 due to head motion 315.

[0082] Generally speaking, the device 305 may generate, display, project, etc. a virtual object to be viewed by the user 310 (e.g., where the virtual object may be displayed based on the user 310 head posture prediction according to the techniques described herein). In some examples, the device 305 may include a transparent surface (e.g., optical glass) so that the virtual object may be displayed on the transparent surface to overlay the virtual object on the real-world object viewed through the transparent surface. Additionally or alternatively, the device 305 may project the virtual object onto the real-world environment. In some cases, the device 305 may include a camera and may display both the real-world object (e.g., as captured by the camera) and the virtual object overlaid on the displayed real-world object. In various examples, the device 305 may include aspects of a virtual reality headset, smart glasses, a live feed video camera, a GPU, one or more sensors (e.g., such as one or more IMUs, image sensors, microphones, etc.), one or more output devices (e.g., such as speakers, displays, smart glasses, etc.), etc.

[0083] In some cases, head movement 315 may include rotation of the user 310's head, translational head movement, and the like. The view 330 of the user 310 may be updated according to the head movement 315. For example, the user 310 may see view 330-A before the head movement 315. In some cases, after the head movement 315, the user 310 may see view 330-B. The extended reality system (e.g., device 305) may render or update virtual objects for display when view 330-A shifts to view 330-B. In some cases, the virtual objects may appear delayed to the user 310 compared to the real-world objects, resulting in visual conflicts. For example, in the absence of the head movement prediction technology described herein, updating the virtual objects in view 330-B from view 330-A may be delayed until the head posture measurement is performed, so that the positioning, orientation, size, etc. of the virtual objects can be updated accordingly. In some cases, the delay may be attributable to system latency (eg, end-to-end system latency between device 305 and a server device), which may be caused by rendering, time warping, or both.

[0084] In some cases, this delay may be referred to as round-trip latency or dynamic registration error. In some cases, the error may be large enough that a user of device 305 may perform head movement 315 before the time at which the gesture measurement is ready for display. Therefore, it may be beneficial to predict head movement 315 so that virtual objects associated with view 330-B may be determined and updated in real time based on the prediction (e.g., pattern) in head movement 315. In some cases, the dynamic registration error may be relatively large. For example, for a system latency of 100 milliseconds (ms) and a moderate head rotation of 50 degrees per second, the dynamic registration error may be 5 degrees. In environments such as distributed extended reality architectures (e.g., Figure 1 In some examples of systems 100 of FIG. 305 , rendering may occur at network 320 . For example, a server device (e.g., server 110 ) may receive a predicted pose from device 305 and may render virtual content based on the predicted pose received from device 305 . In such examples, the latency of the system may increase. User 310 may perceive dynamic registration errors as lag, misalignment of virtual objects, or unnatural appearance, etc.

[0085] In addition, in some cases, network interference may occur when gesture information is sent from device 305 to server device, which can be attributed to overlapping channels in different wireless networks. When there is severe interference, gesture information may not be received on the server device at a fixed pace or time interval (e.g., gesture information is received sporadically across time). If the gesture does not arrive before the rendering call of the server device rendering virtual content, the server device may need to select an old or outdated gesture to render the virtual content. If device 305 uses an outdated gesture to render the virtual content generated by the server device, the virtual content may be visually perceived by the user as translation jitter when displayed, which degrades the user experience.

[0086] As previously noted, systems and techniques are described herein for providing server-based pose prediction (eg, in extended reality (XR) systems). Figure 4 4 is a diagram illustrating an example of a distributed XR system 400 that can implement the techniques described herein. The distributed XR system 400 includes a client device 405 that communicates with a server device 410 via a wired or wireless network. The client device 405 may include an XR device (e.g., an HMD, AR glasses, etc.). In some cases, the client device 405 may be configured to Figure 2 Device 200 or Figure 3 device 305.

[0087] As shown, client device 405 may send pose prediction parameters to server device 410 . Figure 5 is a diagram of a client device 505 illustrating an example configuration of client device 405. Client device 505 includes a tracking system 512 and a transmitter 520 (which may be implemented as a transceiver). In some cases, tracking system 512 is a 6-degree-of-freedom (6DOF) tracker that can track three degrees of rotation data (e.g., including pitch, roll, and yaw) and three degrees of translation data (e.g., horizontal displacement, vertical displacement, and depth displacement relative to a reference point). Tracking system 512 includes one or more sensors 514, a coefficient generator 516, and a pose generator 518.

[0088] In some cases, the set of gesture prediction parameters that may be sent by client device 405 to server device 410 may include raw gesture data (e.g., determined by gesture generator 518), prediction coefficients associated with the raw gesture data (e.g., determined by coefficient generator 516), and a prediction time associated with the time when the raw gesture data was obtained. Figure 5, the raw posture data may be measured or obtained by the client device 505 based on sensor data from one or more sensors 514. For example, the one or more sensors 514 may include one or more IMUs (e.g., accelerometers, gyroscopes, etc.), and the sensor data may include IMU samples from the one or more IMUs. The raw posture data may include 6DOF data representing the posture of the client device 405, such as three-dimensional rotation data (e.g., including pitch, roll, and yaw) and three-dimensional translation data (e.g., horizontal displacement, vertical displacement, and depth displacement relative to a reference point).

[0089] As noted above, the posture prediction data may include prediction coefficients. In some cases, the coefficient generator 516 of the client device 505 may determine the coefficients based on a polynomial used to predict the posture of the client device 505. For example, the prediction coefficients may be bias-corrected IMU coefficients. The coefficient generator 516 may determine the prediction coefficients, such as using a machine learning system or model, as described in more detail herein. The model may take into account the prediction coefficients received from the client device 505. In some cases, the client device 505 may predict the posture of the client device 505 at a future time, which may include determining the prediction coefficients. In such cases, the prediction coefficients determined by the coefficient generator 516 during the prediction of the posture may be sent to the server device 410. In some examples, the model used to determine the prediction coefficients and, in some cases, determine the predicted posture may include a curve-fitted posture based on head rate prediction. For example, the model may fit a curve of past head rate posture changes over time. In some examples, the model may be implemented using a machine learning system or model (e.g., a deep learning network, such as a deep neural network). Reference is made below to Figure 8 and Fig. 9 Further details regarding the determination of prediction coefficients are described.

[0090] The client gesture prediction engine of the server device 410 (e.g., described below) Figure 6B Client posture prediction engine 638 or Figure 7B The client pose prediction engine 738 of the server device 410 may process a pose prediction parameter set (e.g., raw pose data, prediction coefficients, and prediction time) received from the client device 405 to predict or extrapolate the pose of the client device 405. In some examples, the server device 410 may use the current pose, historical pose data, IMU samples, side channel data, or a combination thereof to predict the pose (e.g., corresponding to the head position, head orientation, line of sight such as view 330, etc. of the user 310) of the client device 405 at a future time (e.g., time T, which may be the prediction time) according to the model. As used herein, references to the pose of a client device (e.g., client device 405) and to the head pose of the user of the client device may be used interchangeably.

[0091] In some aspects, server device 410 may process a set of pose prediction parameters (e.g., raw pose data, prediction coefficients, and prediction times) from the received pose prediction parameters to predict or extrapolate poses at fixed time intervals or cadences. In such aspects, server device 410 will continue to observe evenly spaced pose samples as it renders content, even if the reception of pose parameters at server device 410 is intermittent.

[0092] Fig. 6A 6 is a diagram of a server device 610 (as an example configuration of server device 410), which is configured to predict or extrapolate a posture at a fixed time interval 625 based on posture prediction parameters (e.g., raw posture data, prediction coefficients, and prediction time) received from client device 405 or client device 505. Server device 610 includes a receiver 622 (which can be implemented as a transceiver), which is configured to receive one or more posture prediction parameter sets (e.g., raw posture data, prediction coefficients, and prediction time), such as from client device 405 or client device 505. In some examples, client device 405 or 505 sends posture prediction parameters at a specific posture prediction parameter time interval (such as every 11 milliseconds (ms)). Receiver 622 is configured to output updated posture prediction parameters (e.g., updated posture prediction parameter sets) to a posture extrapolation engine 624. As shown, the pose extrapolation engine 624 is configured to extrapolate the predicted pose at each iteration of the fixed time interval 625 (e.g., extrapolate the first pose at the first iteration of the time interval 625, extrapolate the second pose at the second iteration of the time interval 625, etc.). In one illustrative example, the time interval 625 may be 2 ms, in which case the pose extrapolation engine 624 is configured to extrapolate the predicted pose every 2 ms. Figure 6B An illustrative example of a pose extrapolation engine 624, described in further detail below, is shown in FIG.

[0093] Once the gesture is extrapolated or predicted, the gesture extrapolation engine 624 may output the extrapolated gesture to the gesture storage 626. The gesture storage 626 may include a buffer or other storage mechanism configured to store a plurality of gestures. In some cases, the gesture storage 626 may purge outdated or old entries. In an illustrative example, the gesture storage 626 may be configured to store ten gestures at a time, and when a new gesture is received from the gesture extrapolation engine 624, the gesture storage 626 may remove the earliest stored gesture to make room in the gesture storage 626 for the newly extrapolated gesture.

[0094] The rendering engine 628 may obtain or retrieve a gesture from the gesture storage device 626 when it is determined that the virtual content is to be rendered (e.g., based on a rendering call). In an illustrative example, the virtual content may be rendered at a rendering time interval, such as every 11 ms. In some cases, the rendering time interval is an interval longer than the fixed time interval 625, and the gesture is extrapolated by the gesture extrapolation engine 624 at the fixed time interval. Then, the rendering engine 628 may render the virtual content from the predicted gesture retrieved from the gesture storage device 626. The rendering engine 628 may output the rendered virtual content to the encoding engine 630, which may compress (or encode) the virtual content. The transmitter 632 may send the compressed / encoded virtual content as a bitstream to the client device 405 or the client device 505. In some cases, the transmitter 632 may be implemented as a part of a transceiver, which may be the same or different from the transceiver associated with the receiver 622. Client device 405 or client device 505 may receive virtual content from server device 610 (and in some cases decompress or decode the virtual content) and display the virtual content (eg, via a display of client device 405 / 505 ).

[0095] As noted above, Figure 6B 6 is a diagram of an illustrative example of a gesture extrapolation engine 624. As shown, a gesture prediction parameter receiving engine 634 receives updated gesture prediction parameters from a receiver 622. At each iteration of a fixed time interval 625, the gesture prediction parameter receiving engine 634 determines whether a new gesture grouping with an updated set of gesture prediction parameters has been received. For example, as noted above, the gesture prediction parameter time interval may be an interval longer than the fixed time interval 625 (e.g., the gesture prediction parameter time interval may be 11 ms and the fixed time interval 625 may be 2 ms), in which case the gesture extrapolation engine 624 will predict gestures more frequently than it receives updated gesture prediction parameters.

[0096] If the posture prediction parameter receiving engine 634 determines that a new posture grouping has not been received at a given iteration of the fixed time interval 625, the prediction time updating engine 636 will update the prediction time to take into account the amount of time that has passed since the posture extrapolation engine 624 last predicted and extracted the posture. For example, the prediction time updating engine 636 can determine the updated prediction time based on the current time and the previous posture prediction time. In an illustrative example, the updated prediction time can be determined by incrementing the existing prediction time by an amount equal to the current time minus the previous posture prediction time (updated prediction time = current prediction time + (current time - previous posture prediction time)). In an example where the fixed time interval 625 is 2ms, the prediction time updating engine 636 can add a multiple of 2ms to the prediction time at each iteration of the fixed time interval 625 until a new posture prediction parameter grouping is received. The prediction time updating engine 636 can then output the existing posture prediction parameters and the updated prediction time to the client posture prediction engine 638. The client pose prediction engine 638 may then predict the pose of the client device based on the existing pose prediction parameters and an updated prediction time that takes into account any movement that may have occurred in the client device 405 or 505 since the last predicted pose (e.g., at the last iteration of the fixed time interval 625). The client pose prediction engine 638 may output the predicted pose to Fig. 6A A gesture storage device 626 is shown.

[0097] If the posture prediction parameter receiving engine 634 determines that a new posture grouping (e.g., including updated raw posture data, updated prediction coefficients, and updated prediction time) is received at a given iteration of the fixed time interval 625, the posture prediction parameter updating engine 640 may update the posture prediction parameters to those included in the new posture grouping and output the updated posture prediction parameters to the client posture prediction engine 638. The client posture prediction engine 638 may predict the posture of the client device based on the updated posture prediction parameters (such as the updated raw posture data, the updated prediction coefficients, and the updated prediction time). The client posture prediction engine 638 may output the predicted posture to Fig. 6A A gesture storage device 626 is shown.

[0098] Once the client gesture prediction engine 638 predicts the gesture of the client device (e.g., based on existing gesture prediction parameters and an updated prediction time or based on updated gesture prediction parameters), the extrapolated time update engine 642 can update the previous extrapolated time to the present or current time for the next iteration of the fixed time interval 625. The updated previous extrapolated time can be used by the prediction time update engine 636 to update the prediction time if no new gesture packet is received at the next iteration of the fixed time interval 625.

[0099] Return to Figure 4 In some cases, the server device 410 may use a gesture prediction parameter set (e.g., original gesture, prediction coefficient, and prediction time) to predict or extrapolate a gesture when determining that virtual content is to be generated (e.g., in response to receiving a gesture request from a rendering engine of the server device 410, such as at a render call). For example, the server device 410 may use a gesture prediction parameter set (e.g., original gesture data, prediction coefficient, and prediction time) to generate a gesture just before a render call of the rendering engine. In an illustrative example, the rendering engine may request a gesture from a gesture extrapolation engine of the server device 410. The gesture extrapolation engine may predict or extrapolate a gesture based on a recent (e.g., most recent) prediction parameter set (e.g., original gesture data, prediction coefficient, and prediction time), and output the predicted gesture to the rendering engine.

[0100] Fig. 7A 7 is a diagram of a server device 710 (as an example configuration of a server device 410) configured to predict or extrapolate a gesture when determining that virtual content is to be generated. The server device 710 includes a receiver 722 (which can be implemented as a transceiver), which is configured to receive one or more gesture prediction parameter sets (e.g., original gesture data, prediction coefficients, and prediction time), such as from a client device 405 or a client device 505. The receiver 722 is configured to output updated gesture prediction parameters (e.g., updated gesture prediction parameter sets) to a gesture extrapolation engine 724. As shown, the gesture extrapolation engine 724 is configured to extrapolate predicted gestures in response to receiving a gesture request from a rendering engine 728. In some cases, the rendering engine 728 may transmit a gesture request at each rendering call, which corresponds to the time when the rendering engine 728 is configured to render virtual content. Figure 7B An illustrative example of a gesture extrapolation engine 724, described in further detail below, is shown in FIG.

[0101] Once the gesture is extrapolated or predicted, the gesture extrapolation engine 724 may output the extrapolated gesture to the rendering engine 728. The rendering engine 728 may then render virtual content (e.g., based on a render call) using the predicted gesture received from the gesture extrapolation engine 724. The rendering engine 728 may output the rendered virtual content to the encoding engine 730, which may compress (or encode) the virtual content. The transmitter 732 may send the compressed / encoded virtual content as a bitstream to the client device 405 or the client device 505. In some cases, the transmitter 732 may be implemented as part of a transceiver, which may be the same or different from the transceiver associated with the receiver 722. The client device 405 or the client device 505 may receive the virtual content from the server device 710 (and in some cases decompress or decode the virtual content) and display the virtual content (e.g., via a display of the client device 405 / 505).

[0102] As previously pointed out, Figure 7B is a diagram of an illustrative example of a gesture extrapolation engine 724. As shown, a gesture prediction parameter receiving engine 734 receives updated gesture prediction parameters from a receiver 722. Upon receiving a gesture request from a rendering engine 728, the gesture prediction parameter receiving engine 734 determines whether a new gesture grouping with an updated gesture prediction parameter set has been received. In some cases, the gesture prediction parameter time interval for receiving gesture prediction parameters from a client device 405 / 505 (or for sending gesture parameters by a client device 405 / 505) may be an interval longer than a rendering interval (e.g., a gesture prediction parameter time interval may be 11 ms and a rendering interval may be 5 ms), in which case the gesture extrapolation engine 724 will predict gestures more frequently than it receives updated gesture prediction parameters.

[0103] If the gesture prediction parameter receiving engine 734 determines that a new gesture grouping has not been received, the prediction time updating engine 736 may update the prediction time to take into account the amount of time that has passed since the gesture extrapolation engine 724 last predicted and extracted the gesture. For example, if a new gesture grouping is not received, the prediction time updating engine 736 may determine the updated prediction time based on the current time and the previous gesture prediction time. In an illustrative example, the updated prediction time may be determined by incrementing the existing prediction time by an amount equal to the current time minus the previous gesture prediction time (updated prediction time = current prediction time + (current time - previous gesture prediction time)). In an example where the rendering interval is 5 ms, the prediction time updating engine 736 may add multiples of 5 ms to the prediction time whenever a gesture request is received until a new gesture prediction parameter grouping is received. In such cases, when a new gesture grouping is not received, the prediction time updating engine 736 may output the existing gesture prediction parameters and the updated prediction time to the client gesture prediction engine 738. The client pose prediction engine 738 may predict the pose of the client device based on the existing pose prediction parameters and an updated prediction time that takes into account any movement that may have occurred to the client device 405 or 505 since the last predicted pose (e.g., the last time a pose request was received). The client pose prediction engine 738 may output the predicted pose to Fig. 7A Rendering engine 728 is shown.

[0104] If, in response to receiving a gesture request, the gesture prediction parameter receiving engine 734 determines that a new gesture grouping (e.g., including updated raw gesture data, updated prediction coefficients, and updated prediction times) is received, the gesture prediction parameter updating engine 740 may update the gesture prediction parameters to those gesture prediction parameters included in the new gesture grouping. The gesture prediction parameter updating engine 740 may output the updated gesture prediction parameters to the prediction time updating engine 736. The prediction time updating engine 736 may update the prediction time to take into account the time at which the gesture grouping is received. For example, the prediction time updating engine 736 may determine the updated prediction time based on the current time and the time associated with the time at which the updated gesture prediction parameter set is received. In an illustrative example, if the rendering engine 728 requests a gesture from the gesture extrapolation engine 724 at time t but receives the last gesture grouping at t-5ms, the prediction time updating engine 736 may determine the updated prediction time by incrementing the existing prediction time by 5ms. In such cases, when a new gesture packet is received, the prediction time update engine 736 may output updated gesture prediction parameters (e.g., updated raw gesture data, updated prediction coefficients, and updated prediction time) to the client gesture prediction engine 738. The client gesture prediction engine 738 may predict the gesture of the client device based on the updated gesture prediction parameters (such as updated raw gesture data, updated prediction coefficients, and updated prediction time). The client gesture prediction engine 738 may output the predicted gesture to the client device. Fig. 7A Rendering engine 728 is shown.

[0105] Once the client gesture prediction engine 738 predicts the gesture of the client device (e.g., based on the existing gesture prediction parameters and the corresponding updated prediction time or based on the updated gesture prediction parameters and the corresponding updated prediction time), the extrapolated time update engine 742 can update the previous extrapolated time to the present or current time for the next time a gesture request is received. The next time a gesture request is received, the updated previous extrapolated time can be used by the predicted time update engine 736 to update the predicted time.

[0106] As noted above, in some aspects, a model may be used by client device 405 and / or server device 410 (e.g., client pose prediction engine 638 or client pose prediction engine 738) to determine a predicted pose. In some cases, client device 405 may use the model to determine prediction coefficients for pose prediction parameters. In some examples, the model includes a curve fit pose based on head velocity prediction. In some cases, Figure 6B Client posture prediction engine 638 or Figure 7BThe client posture prediction engine 738 can predict the future head posture based on the fitted curve. In some cases, the model can be based on a Kalman filter, an extended Kalman filter (EKF), or a combination with higher order derivatives as part of the internal state of the system. In such cases, the model can implement different filters for rotation (e.g., using an EKF model) and translation (e.g., using a Kalman filter model). However, in some cases, higher order derivatives can cause noise in the system. In other examples, the model can be a time series prediction of posture (e.g., an autoregressive (AR), moving average (MA), or ARMA model). Such examples can produce prediction times greater than 50ms.

[0107] In some cases, Figure 6B Client posture prediction engine 638 or Figure 7B The client pose prediction engine 738 may implement a difference method and exponential smoothing to calculate derivatives (e.g., the derivative of the bias-corrected gyroscope rotation samples may be angular acceleration, etc.). Additionally or alternatively, the client pose prediction engine 638 / 738 may use higher order derivatives to suppress the output to reduce pose jitter. The model for predicting head pose may be calibrated according to Equation 1, where t may refer to a delta difference in time, T may refer to a future time (corresponding to a prediction time), P may indicate a head pose prediction model, and A and B may be system constants.

[0108]

[0109] In some cases, smoothing coefficients and damping factors may be calculated using a grid search (e.g., offline) to increase the accuracy of the calculated pose (e.g., the mean square error (MSE) may be closer to the actual pose or the ground truth pose). This approach may be used for prediction times up to 30 ms to 40 ms (e.g., this may be desirable compared to prediction times greater than 50 ms).

[0110] In some cases, the prediction techniques may benefit from considering range of motion (e.g., the head pose of user 310 may have limited freedom of motion, e.g., due to human anatomy and resulting limitations on head movement). In some cases, the prediction techniques may incorporate range of motion, machine learning models, side channel data, etc.

[0111] The client device 405 can use a deep learning-based technique to determine the prediction coefficient. In an illustrative example, the client device 405 can use a neural network to implement various machine learning techniques. In some machine learning examples, the network can train a machine learning model on a training data set. The training data can be a subset of a larger data set. In some cases, training may involve determining one or more target features in the data set. Subsequently, the model can learn one or more features from the training data (e.g., based on a linear regression technique, such as a linear regression algorithm) and an evaluation metric (such as MSE, precision, accuracy, and recall). In some cases, the evaluation metric can be calculated according to a loss function. Once the model is trained so that it is suitable for the training data (e.g., based on one or more features determined), the model can be used on a data subset different from the training data from a larger data set. The data subset can be referred to as validation data. The validation data set can be used to further adjust the model. In some cases, the model can be used for prediction or decision-making related to features. Examples of machine learning models can implement techniques such as linear regression, logistic regression, decision trees, support vector machines (SVM), naive Bayes, k nearest neighbors, random forests, dimensionality reduction algorithms, gradient boosting algorithms, or any combination thereof.

[0112] Some examples of machine learning models may implement supervised learning techniques as described herein. In some cases, supervised learning techniques may involve a model learning one or more features from a training data set based on an example input-output pair. For example, a learning function may map an input to an output based on an input-output pair. The mapping may involve labeled training data based on the presence of one or more features. Supervised machine learning techniques may use classification, regression, or both to learn one or more features. In some examples, the learning function may implement linear regression, logistic regression, naive Bayes, any similar learning technique, or a combination thereof. In some cases, a machine learning model may implement a semi-supervised learning technique in which a portion of the training data may have a label and another portion may not have a label. For example, a small portion of the labeled training data in the labeled training data may be combined with a large portion of the unlabeled training data in the unlabeled training data for the learning function. Semi-supervised learning techniques may reduce the complexity of the training process (e.g., due to labeling the training data based on features and human supervision).

[0113] Some examples of machine learning models may implement unsupervised learning techniques as described herein. In some cases, unsupervised learning techniques may involve a model learning one or more features from a training data set based on patterns that were not previously detected in the data set. In some cases, unsupervised learning techniques may include grouping or clustering data in a training data set based on commonalities in the data. Additionally or alternatively, unsupervised learning techniques may include associations based on patterns in the training data. Unsupervised learning may implement density estimation in statistical values. For example, unsupervised learning techniques may model probability density on inputs. Some possible learning functions may include k-means for clustering-related applications, a priori techniques for association rule-related applications, any similar learning techniques, or combinations thereof.

[0114] Some examples of machine learning models may implement reinforcement learning techniques as described herein. In some cases, reinforcement learning techniques may implement dynamic programming techniques in which the model may be trained based on feedback and learning functions. For example, the model may use positive or negative feedback to reinforce a value-based reinforcement learning method, a policy-based reinforcement learning method, or a model-based reinforcement learning method. Some examples of reinforcement learning models may include Markov decision processes, Q-learning, any similar learning techniques or models, or combinations thereof.

[0115] In some examples, machine learning may be implemented by a deep learning model such as a deep neural network (e.g., an artificial neural network (ANN)). Figure 8 810, which may include an input layer 815, an output layer 825, and one or more hidden layers 820. Each layer may include a set of interconnected artificial neurons (e.g., nodes). For example, the input layer 815 may include nodes 830-a, 830-b, and 830-c. The hidden layer 820 may include nodes 835-a, 835-b, 835-c, and 835-d. The output layer 825 may include nodes 840-a, 840-b, and 840-c. The connections between nodes may be referred to as edges. These edges may be associated with weights that may be adjusted during the learning process (as part of the training of the deep neural network 810). In some cases, the weights may correspond to an increase or decrease in the value or importance of the nodes at each layer. Neural networks (e.g., ANNs) may be used in applications involving regression and classification. Some additional specific implementations of deep learning may include convolutional neural networks that can be used for computer vision, recursive neural networks that can be used for time series analysis, self-organizing maps that can be used for feature extraction, Boltzmann machines that can be used for recommendation systems, autoencoders that can be used for recommendation systems, and the like.

[0116] In some examples, each layer of the neural network 810 may implement an activation function that may correspond to one or more features in the data set. The activation function may convert the input signal of the node into an output signal. The output signal may be used in the next layer of the neural network (e.g., for an ANN). The activation function may correspond to whether the node can be activated based on calculating the weighted sum of the nodes and adding a bias, which may introduce nonlinearity to the output of the node. In some cases, the activation function may implement a threshold activation function (e.g., a binary step function). In such cases, if the input value of the node is above or below a threshold, the node may be activated. The node may transmit the signal to the next layer. In some cases, the activation function may implement a sigmoid activation function (e.g., a logistic function) to predict a probability as an output. In some examples, the activation function may implement a rectified linear unit (ReLu).

[0117] In some cases, the neural network 810 may implement an activation function for each additional layer of the network (e.g., for the hidden layer 820 between the input layer 815 and the output layer 825). The neural network 810 may also implement a loss function or cost function based on the difference between the actual value and the predicted value. For each layer of the neural network 810, the cost function may be used to adjust the weight of the next input. In some examples, the cost function or loss function may implement an MSE function, which may calculate the square of the difference between the actual value and the predicted value. Once the neural network 810 has been trained (e.g., based on the activation function and adjusting the weights), the neural network 810 may be used for pattern classification, prediction (e.g., for financial analysis), control and optimization (e.g., calibration), or a combination thereof.

[0118] In some examples, input 830 to the machine learning model or neural network 810 (e.g., at input layer 815) may include data from a client device such as client device 405 or 505 for one or more users (e.g., statistics related to the use of an application of the client device, real-time parameters related to future actions, or a combination) or sensor statistics, or both. In some cases, inputs 830-a, 830-b, and 830-c (collectively referred to as input 830) to the neural network 810 may include IMU samples provided by one or more IMU sensors and / or tracking systems (e.g., 6DOF tracking systems). In some cases, input 830 to the machine learning algorithm may include side channel data paired with sensor statistics. In some cases, the neural network 810 may be trained online on application data (e.g., the neural network 810 may be able to learn application-specific patterns to better predict client device / head posture). In some examples, the last layer or last two layers of the neural network 810 (e.g., hidden layer 820) may be fully connected layers trained for samples with high prediction errors.

[0119] The output 840 of the machine learning model may include prediction coefficients or a predicted head pose determined based on the prediction coefficients (e.g., based on input 830 of sensor statistics such as one or more IMU measurements). For example, the machine learning technique may include a model that predicts the pose coefficients based on input 830 of sensor measurements, side channel data, etc. As previously discussed, one or more aspects of the neural network 810 may enable outputting prediction coefficients, which the client device may send to a server device (e.g., server device 410, server device 610, or server device 710), which may then use the prediction coefficients to predict the pose of the client device (e.g., at a future time T) for improved generation, rendering, display, etc. of virtual objects (e.g., so that the virtual object is visible to the user at time T based on the predicted head pose of the user at time T).

[0120] Fig. 9 is a flow chart illustrating an example of a process 900 (e.g., a model) for supporting server-side posture prediction according to aspects of the present disclosure. In some examples, process 900 may implement aspects of a device. For example, a client device (e.g., client device 405 or client device 505) may implement aspects of process 900 for determining prediction coefficients and / or a predicted posture using the determined prediction coefficients. In another example, a server device (e.g., server device 410, server device 610, or server device 710) may implement aspects of process 900 for determining a predicted posture using prediction coefficients received from a client device. In some cases, process 900 may include input 905 and fully connected layers 910 and 930, which may be as described in reference to FIG. Figure 8 An example of the input 830 and hidden layer 820 is shown. For example, the input 905 may include input values, such as sensor data received from one or more sensors 514 (eg, IMU samples from one or more IMUs).

[0121] In some cases, process 900 may have one or more layers that may correspond to one or more functions. For example, process 900 may use input 905 (e.g., at input FC layer 910) to model head pose as a polynomial over a predicted time interval. In general, one or more aspects of process 900 may be implemented to determine prediction coefficients for a future relative head pose prediction function, predict future relative head pose, etc. For example, in some cases, one or more aspects of process 900 may implement machine learning (e.g., which takes into account side channel data, IMU measurements, etc.) to derive a prediction model or prediction function for estimating future relative head pose (e.g., at some time T, so it can be used for virtual object generation, rendering, display, etc. at various times T).

[0122] In some examples, process 900 may implement a Taylor sequence approximation using the se3_exp() function, as shown in Equation 2, to predict relative head pose, where T is the prediction time.

[0123]

[0124] In some cases, the device may determine the prediction coefficients a(t), b(t), and c(t) and the degree of the polynomial using one or more layers of process 900. Where a client device (e.g., client device 405 or client device 505) performs aspects of process 900, the client device may send the prediction coefficients a(t), v(t), and c(t) to a server device (e.g., server device 410, server device 610, or server device 710). Where a server device (e.g., server device 410, server device 610, or server device 710) performs aspects of process 900, the server device may receive the prediction coefficients a(t), b(t), and c(t) from the client device (e.g., client device 405 or client device 505), and the server may predict the relative head pose (corresponding to the pose of the client device).

[0125] In one example, layer 910 may be a fully connected layer, and input 905 may pass through fully connected layer 910. Fully connected layer 910 may convert input 905 (e.g., IMU samples) into a higher dimensional vector. The higher dimensional vector may then be passed into a deep learning architecture at layer 915. In some cases, if input 905 is sequential, process 900 may implement a long short-term memory (LSTM) based architecture at layer 915. The LSTM based architecture may use the higher dimensional vector from layer 910, the history of past sensor statistics 920 (e.g., IMU samples, i.e., h in ) or both. The output of the LSTM architecture at layer 915 (e.g., h out ) can be saved at 925 (e.g., to be applied to the next input) and can be passed through a fully connected layer 930. The fully connected layer 930 can convert the output of the LSTM at layer 915 into prediction coefficients for process 900, such as prediction coefficients 935 (e.g., coefficients a(t), b(t), and c(t)).

[0126] In some cases, once process 900 has prediction coefficients 935 and prediction time T, the polynomial at Equation 2 may be used at fixed layer 940 to determine a predicted head pose by a server device (and in some cases a client device) (e.g., corresponding to input 905, shown at 945). In some cases, if the prediction time is relatively short (e.g., if the head pose is to be predicted in the relatively near future or at less time T), the approximation may have a low degree polynomial. In other cases, if the prediction time is relatively long (e.g., if the head pose is to be predicted in the relatively farther future or at more time T), the approximation may have a higher degree (e.g., higher order) polynomial. In some cases, a neural network (e.g., Figure 8 The neural network 810 of the embodiment of the present invention can be trained by calculating the loss at the loss calculation layer 950, which can be the difference between the predicted pose from the fixed layer 940 and the ground truth pose (e.g., the pose from the training data). In some cases, the process 900 can propagate the loss at the loss calculation layer 950 to the network 120 (e.g., to train the process 900).

[0127] In some examples, process 900 may use the se(3) function to predict the translation, rotation, or both of the head pose corresponding to the pose of the client device. For example, the rotation model may have a rotation axis with respect to the unit 3 parameters related to the rotation angle (θ). Additionally or alternatively, the translation model may have 3 parameters (e.g., horizontal, vertical, and depth). The combined rotation model and translation model may both define a helical axis. Rotation, translation, or both around the helical axis may correspond to rigid body motion in three dimensions.

[0128] In some cases, process 900 may use the bias-corrected IMU coefficients to determine a predicted head pose corresponding to the pose of the client device. For example, process 900 may use Equation 3 as an input to a deep learning architecture layer 915 (e.g., an LSTM architecture), where ω b (t) is the angular velocity in the main frame, ω g (t) is the angular velocity in the gyroscope frame, R bg is the relative orientation between the subject and the gyroscope, b g is the gyroscope bias, b a is the accelerometer bias, A g and A a is the non-orthogonality and scaling factor, α b (t) is the linear acceleration in the body frame, and y b is the gravity in the main frame.

[0129]

[0130] Equation 3 may correspond to bias-corrected acceleration and gyroscope samples. In some cases, given a history of sensor statistics 920 (e.g., corresponding to hidden states) and inputs from Equation 3 (e.g., (ω g (t),α b (t)), the relative temporal pose (e.g., in se(3), for the prediction time T) can be calculated according to Equation 4. For example, Equation 4 can be used (e.g., by the network) to predict the coefficients a for the rotation and translation at time t r (t).

[0131]

[0132] In some cases, the loss function at the loss calculation layer 950 can be calculated based on the MSE loss of the relative pose within a time interval (e.g., 100ms). For example, if there are 10 samples within the prediction time of 100ms, the average MSE can be calculated over all samples relative to the ground truth pose in se(3) (e.g., by calculating the L2 norm of the change in pose in se(3)). Additionally, the loss function can have a term that takes into account jitter.

[0133] In some cases, process 900 may include uncertainty in the prediction. For example, uncertainty may be included as a function of prediction error. In some cases, if the prediction error is low, there may be a direct correlation with uncertainty. Prediction uncertainty can be used to infer information related to rendering and processing time. In some cases, if the uncertainty is low, additional rendering will also be relatively low. That is, in some cases, additional rendering may occur to compensate for the pose prediction error, so if the pose prediction error is relatively low, the uncertainty may also be low. Therefore, prediction uncertainty can be used to determine the amount of additional rendering.

[0134] In some cases, side information such as application information or data associated with the extended reality system may be used in addition or as an alternative to training or updating process 900. For example, machine learning-based pose predictions (e.g., predictions generated by process 900) may learn patterns of head pose movement. In some cases, process 900 may take into account the limited range of motion of the human head (e.g., limited degree of rotation, limited degrees of freedom, etc.). In some cases, sudden changes in head positioning or head movement may lead to uncertainty in the model. Therefore, it may be beneficial to use application information or data to predict sudden changes.

[0135] In some cases, the application information or data may correspond to a game or application, content. For example, there may be an event that may cause a sudden change in the user's head positioning (e.g., in a game). The application information or data can be used to determine whether the user is likely to change the head positioning. Similarly, process 900 can use eye tracking data to calculate what the user may be looking at, and then use this information to predict sudden changes in head positioning. For example, if the user is chasing or tracking a moving ball in augmented reality, if the rendered ball changes direction, the user's head posture may also change direction to follow the ball. Additionally or alternatively, process 900 can use body posture estimation output to determine capabilities related to human range of motion. In some cases, process 900 can implement posture detection and use the results to determine the range of motion. For example, if the user is sitting in a chair, the range of motion of the head posture may be different from standing. In some examples, audio data can be used to determine or predict the user's head posture or head movement (e.g., rotating toward the sound source). In some examples, the accuracy of head posture prediction can be improved using application information or data (e.g., for sudden changes in head posture).

[0136] In some examples, a deep learning-based posture prediction model trained on offline data (e.g., trained based on process 900) may be additionally trained with application information or data. In such examples, process 900 may take into account patterns associated with an application or game, which may correspond to improved accuracy in posture prediction. In some cases, process 900 may implement application information or data for training at the end of the training process. For example, process 900 may use application information or data in a fully connected layer (e.g., the last one or two fully connected layers, such as fully connected layer 930) for inputs with high prediction errors. That is, application information or data may be indirectly incorporated into process 900. Therefore, models for different applications (e.g., games) may differ depending on the content or pattern in the application.

[0137] In some cases, the deep learning-based posture prediction model may be directly incorporated into the application information or data used for posture prediction. For example, input 905 may include information or data for posture prediction other than sensor statistics (e.g., game application state and IMU samples). The deep learning basic posture prediction model may be directly incorporated into the application information or data, indirectly incorporated into the application information or data, or both of the above. In some cases, the system may use the output and application information or data from the deep learning-based posture prediction model (e.g., process 900) as input to the linear regression model. The weights of the deep learning model may correspond to one or more learning techniques or models. In some examples, the system may use uncertainty to determine whether to incorporate application information or data. For example, if the uncertainty of the deep learning-based posture prediction model (e.g., process 900) is high, the system may use application information or data (e.g., game state) to predict the head posture. If the uncertainty of the deep learning-based posture prediction model is low, the system may not use the application information or data in the model.

[0138] Fig.10 An example of a process 1000 for generating virtual content at a first device of a distributed extended reality system is illustrated. According to aspects described herein, the first device may include a server device (e.g., an edge-based or cloud-based server, a personal computer acting as a server device, a mobile device such as a mobile phone acting as a server device, an XR device acting as a server device, a network router, or other device acting as a server). Illustrative examples of server devices include server device 410, Fig. 6A The server device 610 and Fig. 7A Server device 710. In some examples, a component (eg, a chipset, a processor, a memory, any combination thereof, and / or other components) may perform one or more of the operations of process 1000.

[0139] At box 1005, the first device (or its component) may receive one or more gesture prediction parameter sets from the second device associated with the distributed extended reality system. Each corresponding gesture prediction parameter set from one or more gesture prediction parameter sets may include a corresponding prediction coefficient and a corresponding prediction time associated with the corresponding original gesture data. In some aspects, the corresponding prediction coefficient may be generated using a machine learning system (e.g., by the second device). In some cases, each corresponding gesture prediction parameter set from one or more gesture prediction parameter sets may also include corresponding original gesture data. In some examples, the corresponding original gesture data included in each corresponding gesture prediction parameter set is measured based on a corresponding inertial measurement unit associated with at least one sensor of the second device.

[0140] At block 1010, the first device (or a component thereof) may predict one or more postures of the second device based on one or more posture prediction parameter sets. In some aspects, the predicted one or more postures of the second device include at least one future posture of the second device at a future time relative to a prediction time included in a posture prediction parameter set from the one or more posture prediction parameter sets. In some cases, at least one future posture includes a relative head posture rotation at the future time and a relative head posture translation at the future time.

[0141] In some aspects, to predict one or more postures of the second device based on one or more posture prediction parameter sets, the first device (or a component thereof) may model the relative posture of the second device as a time polynomial within a prediction interval based at least in part on raw posture data and prediction coefficients included in a posture prediction parameter set from the one or more posture prediction parameter sets. The first device (or a component thereof) may determine at least one future posture of the second device based on the relative posture and the prediction time.

[0142] In some aspects, to predict one or more postures of the second device, the first device (or a component thereof) may predict multiple postures of the second device (e.g., relative to the first device) at fixed time intervals. Fig. 6A and Figure 6B The first device (or a component thereof) may predict each of the plurality of gestures at fixed time intervals based on at least one corresponding gesture prediction parameter in one or more gesture prediction parameter sets.

[0143] In some cases, the first device (or a component thereof) may determine that a set of posture prediction parameters is received at an iteration of a fixed time interval. The first device (or a component thereof) may predict the posture of the second device based on the set of posture prediction parameters for the iteration of the fixed time interval.

[0144] In some cases, the first device (or its component) may determine that a posture prediction parameter set is not received at an iteration of a fixed time interval. The first device (or its component) may determine an updated prediction time based on the current time and the previous posture prediction time based on determining that a posture prediction parameter set is not received at an iteration of a fixed time interval. The first device (or its component) may predict the posture of the second device based on the previously received posture prediction parameter set and the updated prediction time for the iteration of the fixed time interval.

[0145] In some aspects, to predict one or more postures of the second device, the first device (or a component thereof) may determine that virtual content is to be generated, and may predict a posture of the second device (e.g., such as relative to the first device) based on the determination that virtual content is to be generated. Fig. 7A and Figure 7BFor example, the first device (or a component thereof) may be based on a display from a rendering engine (e.g., Fig. 7A The rendering engine 730) determines the gesture request of the virtual content to be generated.

[0146] In some cases, the first device (or its component) may determine that a gesture prediction parameter set is received at a first time associated with the time when the virtual content is to be generated. The first device (or its component) may determine an updated prediction time based on the current time and a second time associated with the time when the gesture prediction parameter set is received based on determining that the gesture prediction parameter set is received at the first time. The first device (or its component) may further predict the gesture of the second device based on the gesture prediction parameter set and the updated prediction time. In some examples, the first time is equal to the second time. In some examples, the first time is different from the second time.

[0147] In some cases, the first device (or a component thereof) may determine that a gesture prediction parameter set is not received at a time associated with the time when the virtual content is to be generated. The first device (or a component thereof) may determine an updated prediction time based on the current time and a previous gesture prediction time based on determining that a gesture prediction parameter set is not received at a time associated with the time when the virtual content is to be generated. The first device (or a component thereof) may further predict the gesture of the second device based on the previously received gesture prediction parameter set and the updated prediction time.

[0148] In some aspects, the first device (or its components) may further predict one or more gestures of the first device based on the common clock reference. In some aspects, the first device (or its components) may further predict one or more gestures of the first device based on a predicted time at which at least one frame will be rendered.

[0149] At block 1015, the first device (or a component thereof) may generate virtual content based on a gesture of the predicted one or more gestures of the first device, as described herein. At block 1020, the first device (or a component thereof) may send the virtual content to the second device (e.g., as an encoded bitstream).

[0150] Fig.11 An example of a process 1100 for processing virtual data at a first device of a distributed extended reality system is illustrated. According to aspects described herein, the first device may include a client device (e.g., an extended reality device acting as a client device, a mobile device such as a mobile phone acting as a client device, a personal computer acting as a client device, or other device acting as a client device). Illustrative examples of client devices include Figure 4 Client device 405 and Figure 5Client device 505. In some examples, a component (e.g., a chipset, a processor, a memory, any combination thereof, and / or other components) may perform one or more of the operations of process 1100.

[0151] At box 1105, the first device (or its component) may send one or more gesture prediction parameter sets to the second device associated with the distributed extended reality system. Each corresponding gesture prediction parameter set from the one or more gesture prediction parameter sets may include a corresponding prediction coefficient and a corresponding prediction time associated with the corresponding original gesture data. In some aspects, the first device may use a machine learning system to generate prediction coefficients. In some cases, each corresponding gesture prediction parameter set from the one or more gesture prediction parameter sets may also include corresponding original gesture data. In some cases, the corresponding original gesture data included in each corresponding gesture prediction parameter set is measured based on a corresponding inertial measurement unit associated with at least one sensor of the first device. In some aspects, the one or more predicted gestures of the first device include at least one future gesture of the first device at a future time relative to the prediction time included in the gesture prediction parameter set from the one or more gesture prediction parameter sets. In some cases, at least one future gesture includes a relative head gesture rotation at a future time and a relative head gesture translation at a future time.

[0152] At block 1110, the first device (or a component thereof) may receive virtual content (e.g., as an encoded bitstream) based on a gesture in one or more predicted gestures of the first device from the second device, the one or more predicted gestures being predicted based on one or more gesture prediction parameter sets sent to the second device. At block 1115, the first device (or a component thereof) may display one or more virtual objects based at least in part on the received virtual content. In some examples, the first device (or a component thereof) may determine a relative display positioning of one or more virtual objects based at least in part on at least one future gesture of a user of the first device. The first device (or a component thereof) may display one or more virtual objects based at least in part on the relative display positioning.

[0153] In some cases, a device or apparatus configured to perform the operations of process 1000 and / or process 1100 may include a processor, microprocessor, microcomputer, or other component of a device configured to perform the steps of process 1000 and / or process 1100. In some examples, such a device or apparatus may include one or more sensors configured to capture image data and / or other sensor measurements. In some examples, such a computing device or apparatus may include one or more sensors and / or cameras configured to capture one or more images or videos. In some cases, such a device or apparatus may include a display for displaying an image. In some examples, one or more sensors and / or cameras are separated from the device or apparatus, in which case the device or apparatus receives the sensed data. Such a device or apparatus may also include a network interface configured to convey data.

[0154] Components of a device or apparatus configured to perform one or more operations of process 1000 and / or process 1100 may be implemented in circuits. For example, a component may include an electronic circuit or other electronic hardware, and / or may be implemented using an electronic circuit or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein. The computing device may also include a display (as an example of an output device or as a supplement to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0155] Process 1000 and process 1100 are illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0156] Additionally, the processes described herein (e.g., process 1000 and / or process 1100) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, implemented by hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0157] Fig.12 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig.12 An example of a computing system 1200 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 1205. The connection 1205 may be a physical connection using a bus, or a direct connection into the processor 1210, such as in a chipset architecture. The connection 1205 may also be a virtual connection, a networked connection, or a logical connection.

[0158] In some aspects, computing system 1200 is a distributed system in which the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components that each perform some or all of the functionality that the component is described for. In some aspects, each component may be a physical or virtual device.

[0159] The example system 1200 includes at least one processing unit (CPU or processor) 1210 and connections 1205 that couple various system components including system memory 1215, such as read-only memory (ROM) 1220 and random access memory (RAM) 1225, to the processor 1210. The computing system 1200 may include a cache 1211 of high-speed memory directly connected to, close to, or integrated as part of the processor 1210.

[0160] Processor 1210 may include any general purpose processor and hardware or software services such as services 1232, 1234, and 1236 stored in storage device 1230 configured to control processor 1210, as well as special purpose processors where software instructions are incorporated into the actual processor design. Processor 1210 may be essentially a fully self-contained computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0161] To enable user interaction, the computing system 1200 includes an input device 1245 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The computing system 1200 may also include an output device 1235, which may be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1200. The computing system 1200 may include a communication interface 1240, which may generally govern and manage user input and system output.

[0162] The communication interface can perform or facilitate the use of wired and / or wireless transceivers to receive and / or send wired or wireless communications, including using audio jacks / plugs, microphone jacks / plugs, universal serial bus (USB) ports / plugs, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transmission, Low power (BLE) wireless signal transmission, Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, WLAN signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / Long Term Evolution (LTE) cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof.

[0163] The communication interface 1240 may also include one or more GNSS receivers or transceivers for determining the location of the computing system 1200 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the Global Positioning System (GPS) of the United States, the Global Navigation Satellite System (GLONASS) of Russia, the BeiDou Navigation Satellite System (BDS) of China, and the Galileo GNSS of Europe. There is no limitation to operating on any particular hardware arrangement, and thus the underlying features herein may be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0164] The storage device 1230 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium capable of storing data that can be accessed by a computer, such as a cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a floppy disk, a hard disk, a magnetic tape, a magnetic stripe / magnetic stripe, any other magnetic storage medium, flash memory, a memristor memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) disc, a rewritable compact disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, a Europay, MasterCard and Visa (EMV) chip, a Subscriber Identity Module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a RAM, a static RAM (SRAM), a dynamic RAM (DRAM), a ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASH EPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0165] Storage device 1230 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1210, causes the system to perform functions. In some aspects, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 1210, connector 1205, output device 1235, etc. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and does not include carrier waves and / or transient electronic signals propagated wirelessly or over a wired connection.

[0166] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memory, memory, or memory devices. Codes and / or machine-executable instructions may be stored on computer-readable media, which may represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0167] In some aspects, computer readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media expressly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0168] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can also be practiced without these specific details. For the sake of clarity of explanation, in some cases, the present technology may be presented as including a separate functional block, which includes a device, device component, step or routine in a method embodied in a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing various aspects.

[0169] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many of the operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operation of the process is completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to a calling function or a main function.

[0170] The processes and methods according to the above examples can be implemented using computer executable instructions stored or otherwise obtained from computer readable media. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or functional group. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions (such as assembly language), firmware, source code. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memories, USB devices with non-volatile memory, networked storage devices, etc.

[0171] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed in a single device.

[0172] Instructions, media for transmitting such instructions, computing resources for executing the instructions, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0173] In the above description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, various aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that, in alternative aspects, each method can be performed in a different order than described.

[0174] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0175] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0176] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component and / or any component is in direct or indirect communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0177] Claim language or other language in this disclosure that states "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that states "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language that states "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that states "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0178] Various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, frames, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0179] The technology described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technology may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses, including applications in wireless communication device mobile phones and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the technology may be implemented at least in part by a computer-readable data storage medium including a program code, which includes instructions for executing one or more of the methods, algorithms, and / or operations described above when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0180] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein.

[0181] Illustrative aspects of the present disclosure include:

[0182] Aspect 1. A method for generating virtual content at a first device of a distributed extended reality system, comprising: receiving one or more posture prediction parameter sets from a second device associated with the distributed extended reality system, wherein each corresponding posture prediction parameter set from the one or more posture prediction parameter sets includes a corresponding prediction coefficient and a corresponding prediction time associated with corresponding original posture data; predicting one or more postures of the second device based on the one or more posture prediction parameter sets; generating virtual content based on a posture among the predicted one or more postures of the first device; and sending the virtual content to the second device.

[0183] Aspect 2. The method according to aspect 1, wherein each corresponding posture prediction parameter set from the one or more posture prediction parameter sets further includes the corresponding original posture data.

[0184] Aspect 3. A method according to aspect 2, wherein the respective raw posture data included in each respective posture prediction parameter set is based on respective inertial measurement unit measurements associated with at least one sensor of the second device.

[0185] Aspect 4. A method according to any one of Aspects 1 to 3, wherein the corresponding prediction coefficients are generated using a machine learning system.

[0186] Aspect 5. A method according to any one of Aspects 1 to 4, wherein the predicted one or more postures of the second device include at least one future posture of the second device at a future time relative to a prediction time included in a posture prediction parameter set from the one or more posture prediction parameter sets.

[0187] Aspect 6. The method according to aspect 5, wherein the at least one future pose comprises a relative head pose rotation at the future time and a relative head pose translation at the future time.

[0188] Aspect 7. A method according to any one of Aspects 4 or 5, wherein predicting the one or more postures of the second device based on the one or more posture prediction parameter sets includes: modeling the relative posture of the second device as a time polynomial within a prediction interval based at least in part on original posture data and prediction coefficients included in a posture prediction parameter set from the one or more posture prediction parameter sets; and determining at least one future posture of the second device based on the relative posture and the prediction time.

[0189] Aspect 8. A method according to any one of Aspects 1 to 7, wherein predicting the one or more postures of the second device includes: predicting multiple postures of the second device at fixed time intervals, each of the multiple postures being predicted at the fixed time interval based on at least one corresponding posture prediction parameter in the one or more posture prediction parameter sets.

[0190] Aspect 9. The method according to Aspect 8 further includes: determining that a posture prediction parameter set is received at an iteration of the fixed time interval; and predicting the posture of the second device based on the posture prediction parameter set for the iteration of the fixed time interval.

[0191] Aspect 10. The method according to Aspect 8 also includes: determining that a posture prediction parameter set is not received at an iteration of the fixed time interval; determining an updated prediction time based on the current time and a previous posture prediction time based on determining that the posture prediction parameter set is not received at the iteration of the fixed time interval; and predicting the posture of the second device based on the previously received posture prediction parameter set and the updated prediction time for the iteration of the fixed time interval.

[0192] Aspect 11. A method according to any one of Aspects 1 to 10, wherein predicting the one or more postures of the second device comprises: determining that the virtual content is to be generated; and predicting the posture of the second device based on determining that the virtual content is to be generated.

[0193] Aspect 12. The method according to aspect 11, wherein determining that the virtual content is to be generated is based on a gesture request from a rendering engine.

[0194] Aspect 13. The method according to any one of Aspects 11 or 12 further includes: determining that a posture prediction parameter set is received at a first time associated with the time when the virtual content is to be generated; determining an updated prediction time based on the current time and a second time associated with the time when the posture prediction parameter set is received based on determining that the posture prediction parameter set is received at the first time; and predicting the posture of the second device based on the posture prediction parameter set and the updated prediction time.

[0195] Aspect 14. The method according to aspect 13, wherein the first time is equal to the second time.

[0196] Aspect 15. The method according to aspect 13, wherein the first time is different from the second time.

[0197] Aspect 16. The method according to any one of Aspects 11 or 12 further includes: determining that a posture prediction parameter set is not received at a time associated with the time when the virtual content is to be generated; determining an updated prediction time based on the current time and a previous posture prediction time based on determining that the posture prediction parameter set is not received at the time associated with the time when the virtual content is to be generated; and predicting the posture of the second device based on the previously received posture prediction parameter set and the updated prediction time.

[0198] Aspect 17. The method according to any one of aspects 1 to 16 further includes: predicting the one or more postures of the first device further based on a common clock reference.

[0199] Aspect 18. The method according to any one of aspects 1 to 17 further comprises: predicting the one or more postures of the first device further based on a predicted time when at least one frame will be rendered.

[0200] Aspect 19. A first device for generating virtual content at a first device in a distributed extended reality system, the first device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory, the at least one processor being configured to: receive one or more posture prediction parameter sets from a second device associated with the distributed extended reality system, wherein each corresponding posture prediction parameter set from the one or more posture prediction parameter sets comprises a corresponding prediction coefficient and a corresponding prediction time associated with corresponding original posture data; predict one or more postures of the second device based on the one or more posture prediction parameter sets; generate virtual content based on a posture in the predicted one or more postures of the first device; and send the virtual content to the second device.

[0201] Aspect 20. The first device according to aspect 19, wherein each respective pose prediction parameter set from the one or more pose prediction parameter sets further comprises the respective raw pose data.

[0202] Aspect 21. The first device according to aspect 20, wherein the respective raw posture data included in each respective set of posture prediction parameters is based on respective inertial measurement unit measurements associated with at least one sensor of the second device.

[0203] Aspect 22. The first device according to any one of aspects 19 to 21, wherein the corresponding prediction coefficients are generated using a machine learning system.

[0204] Aspect 23. A first device according to any one of Aspects 19 to 22, wherein the predicted one or more postures of the second device include at least one future posture of the second device at a future time relative to a prediction time included in a posture prediction parameter set from the one or more posture prediction parameter sets.

[0205] Aspect 24. The first device according to aspect 23, wherein the at least one future pose comprises a relative head pose rotation at the future time and a relative head pose translation at the future time.

[0206] Aspect 25. A first device according to any one of Aspects 23 or 24, wherein in order to predict the one or more postures of the second device based on the one or more posture prediction parameter sets, the at least one processor is configured to: model the relative posture of the second device as a time polynomial within a prediction interval based at least in part on the original posture data and prediction coefficients included in the posture prediction parameter sets from the one or more posture prediction parameter sets; and determine the at least one future posture of the second device based on the relative posture and the prediction time.

[0207] Aspect 26. A first device according to any one of Aspects 19 to 25, wherein in order to predict the one or more postures of the second device, the at least one processor is configured to: predict multiple postures of the second device at fixed time intervals, each of the multiple postures being predicted at the fixed time interval based on at least one corresponding posture prediction parameter in the one or more posture prediction parameter sets.

[0208] Aspect 27. A first device according to Aspect 26, wherein the at least one processor is configured to: determine that a set of posture prediction parameters is received at an iteration of the fixed time interval; and predict the posture of the second device based on the set of posture prediction parameters for the iteration of the fixed time interval.

[0209] Aspect 28. A first device according to Aspect 26, wherein the at least one processor is configured to: determine that a posture prediction parameter set is not received at an iteration of the fixed time interval; determine an updated prediction time based on a current time and a previous posture prediction time based on determining that the posture prediction parameter set is not received at the iteration of the fixed time interval; and predict the posture of the second device based on the previously received posture prediction parameter set and the updated prediction time for the iteration of the fixed time interval.

[0210] Aspect 29. A first device according to any one of Aspects 19 to 28, wherein in order to predict the one or more postures of the second device, the at least one processor is configured to: determine that the virtual content is to be generated; and predict the posture of the second device based on determining that the virtual content is to be generated.

[0211] Aspect 30. The first device according to aspect 29, wherein the at least one processor is configured to determine that the virtual content is to be generated based on a gesture request from a rendering engine.

[0212] Aspect 31. A first device according to any one of Aspects 29 or 30, wherein the at least one processor is configured to: determine that a posture prediction parameter set is received at a first time associated with the time when the virtual content is to be generated; determine an updated prediction time based on a current time and a second time associated with the time when the posture prediction parameter set is received based on determining that the posture prediction parameter set is received at the first time; and predict the posture of the second device based on the posture prediction parameter set and the updated prediction time.

[0213] Aspect 32. The first device according to aspect 31, wherein the first time is equal to the second time.

[0214] Aspect 33. The first device according to aspect 31, wherein the first time is different from the second time.

[0215] Aspect 34. A first device according to any one of Aspects 29 or 30, wherein the at least one processor is configured to: determine that a posture prediction parameter set is not received at a time associated with the time when the virtual content is to be generated; determine an updated prediction time based on a current time and a previous posture prediction time based on determining that the posture prediction parameter set is not received at the time associated with the time when the virtual content is to be generated; and predict the posture of the second device based on the previously received posture prediction parameter set and the updated prediction time.

[0216] Aspect 35. The first device according to any one of aspects 19 to 34, wherein the at least one processor is configured to: predict the one or more gestures of the first device further based on a common clock reference.

[0217] Aspect 36. The first device according to any one of Aspects 19 to 35, wherein the at least one processor is configured to: predict the one or more postures of the first device further based on a predicted time at which at least one frame will be rendered.

[0218] Aspect 37. A first device for processing virtual data in a distributed extended reality system, the first device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory, the at least one processor being configured to: send one or more posture prediction parameter sets to a second device associated with the distributed extended reality system, wherein each corresponding posture prediction parameter set from the one or more posture prediction parameter sets comprises a corresponding prediction coefficient and a corresponding prediction time associated with corresponding original posture data; receive virtual content based on postures in the one or more predicted postures of the first device from the second device, the one or more predicted postures being predicted based on the one or more posture prediction parameter sets sent to the second device; and display one or more virtual objects based at least in part on the received virtual content.

[0219] Aspect 38. The first device according to aspect 37, wherein each corresponding set of posture prediction parameters from the one or more sets of posture prediction parameters also includes the corresponding raw posture data.

[0220] Aspect 39. The first device according to aspect 38, wherein the respective raw posture data included in each respective set of posture prediction parameters is based on respective inertial measurement unit measurements associated with at least one sensor of the first device.

[0221] Aspect 40. A first device according to any one of Aspects 37 to 39, wherein the one or more predicted postures of the first device include at least one future posture of the first device at a future time relative to a predicted time included in a posture prediction parameter set from the one or more posture prediction parameter sets.

[0222] Aspect 41. The first device according to aspect 40, wherein the at least one future pose comprises a relative head pose rotation at the future time and a relative head pose translation at the future time.

[0223] Aspect 42. A first device according to any one of Aspects 40 or 41, wherein the at least one processor is configured to: determine the relative display position of the one or more virtual objects based at least in part on the at least one future gesture of the user of the first device; and display the one or more virtual objects based at least in part on the relative display position.

[0224] Aspect 43. A method for processing virtual data at a first device in a distributed extended reality system, comprising: sending one or more posture prediction parameter sets to a second device associated with the distributed extended reality system, wherein each corresponding posture prediction parameter set from the one or more posture prediction parameter sets includes a corresponding prediction coefficient and a corresponding prediction time associated with corresponding original posture data; receiving virtual content based on postures in the one or more predicted postures of the first device from the second device, the one or more predicted postures being predicted based on the one or more posture prediction parameter sets sent to the second device; and displaying one or more virtual objects based at least in part on the received virtual content.

[0225] Aspect 44. The method according to aspect 43, wherein each corresponding posture prediction parameter set from the one or more posture prediction parameter sets also includes the corresponding original posture data.

[0226] Aspect 45. A method according to aspect 44, wherein the respective raw posture data included in each respective set of posture prediction parameters is based on respective inertial measurement unit measurements associated with at least one sensor of the first device.

[0227] Aspect 46. A method according to any one of Aspects 43 to 45, wherein the one or more predicted postures of the first device include at least one future posture of the first device at a future time relative to a predicted time included in a posture prediction parameter set from the one or more posture prediction parameter sets.

[0228] Aspect 47. The method according to aspect 46, wherein the at least one future pose comprises a relative head pose rotation at the future time and a relative head pose translation at the future time.

[0229] Aspect 48. The method according to any one of Aspects 46 or 47 further includes: determining the relative display position of the one or more virtual objects based at least in part on the at least one future gesture of the user of the first device; and displaying the one or more virtual objects based at least in part on the relative display position.

[0230] Aspect 49. Provided is a non-transitory computer-readable medium having instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform operations according to any one of Aspects 1 to 36.

[0231] Aspect 50. A first device for generating virtual content in a distributed extended reality system, the first device comprising: one or more components for performing the operations according to any one of Aspects 1 to 36.

[0232] Aspect 51. Provided is a non-transitory computer-readable medium having instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform operations according to any one of aspects 37 to 48.

[0233] Aspect 52. A first device for generating virtual content in a distributed extended reality system, the first device comprising: one or more components for performing the operations according to any one of Aspects 37 to 48.

Claims

1. A first device for generating virtual content in a distributed extended reality system, the first device comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: receiving one or more gesture prediction parameter sets from a second device associated with the distributed extended reality system, wherein each respective gesture prediction parameter set from the one or more gesture prediction parameter sets comprises a respective prediction coefficient and a respective prediction time associated with respective raw gesture data; predicting one or more postures of the second device based on the one or more sets of posture prediction parameters; generating virtual content based on a gesture of the predicted one or more gestures of the first device; as well as The virtual content is sent to the second device. 2 . The first device of claim 1 , wherein each respective set of gesture prediction parameters from the one or more sets of gesture prediction parameters further comprises the respective raw gesture data.

3. The first device of claim 2, wherein the respective raw posture data included in each respective set of posture prediction parameters is based on respective inertial measurement unit measurements associated with at least one sensor of the second device.

4. The first device of claim 1 , wherein the predicted one or more postures of the second device include at least one future posture of the second device at a future time relative to a prediction time included in a posture prediction parameter set from the one or more posture prediction parameter sets. 5 . The first device of claim 4 , wherein the at least one future pose comprises a relative head pose rotation at the future time and a relative head pose translation at the future time.

6. The first device of claim 4, wherein to predict the one or more postures of the second device based on the one or more posture prediction parameter sets, the at least one processor is configured to: modeling the relative pose of the second device as a time polynomial within a prediction interval based at least in part on raw pose data and prediction coefficients included in a pose prediction parameter set from the one or more pose prediction parameter sets; and The at least one future pose of the second device is determined based on the relative pose and the predicted time.

7. The first device of claim 1, wherein the corresponding prediction coefficients are generated using a machine learning system.

8. The first device of claim 1 , wherein to predict the one or more gestures of the second device, the at least one processor is configured to: A plurality of postures of the second device are predicted at fixed time intervals, each posture of the plurality of postures being predicted at the fixed time intervals based on at least one corresponding posture prediction parameter of the one or more posture prediction parameter sets.

9. The first device of claim 8, wherein the at least one processor is configured to: Determining that a set of posture prediction parameters is received at an iteration of the fixed time interval; and For the iterations of the fixed time interval, a posture of the second device is predicted based on the set of posture prediction parameters.

10. The first device of claim 8, wherein the at least one processor is configured to: determining that a set of posture prediction parameters is not received at an iteration of the fixed time interval; determining an updated prediction time based on a current time and a previous pose prediction time based on determining that the pose prediction parameter set is not received at the iteration of the fixed time interval; and For the iterations of the fixed time interval, a posture of the second device is predicted based on a previously received set of posture prediction parameters and the updated prediction time.

11. The first device of claim 1 , wherein to predict the one or more gestures of the second device, the at least one processor is configured to: determining that the virtual content is to be generated; and The pose of the second device is predicted based on determining that the virtual content is to be generated. 12 . The first device of claim 11 , wherein the at least one processor is configured to determine that the virtual content is to be generated based on a gesture request from a rendering engine.

13. The first device of claim 11, wherein the at least one processor is configured to: determining that a set of gesture prediction parameters is received at a first time associated with a time at which the virtual content is to be generated; determining an updated prediction time based on a determination that the set of gesture prediction parameters is received at the first time and based on a current time and a second time associated with a time when the set of gesture prediction parameters is received; as well as The pose of the second device is predicted based on the set of pose prediction parameters and the updated prediction time. The first device of claim 13 , wherein the first time is equal to the second time. The first device of claim 13 , wherein the first time is different from the second time.

16. The first device of claim 11, wherein the at least one processor is configured to: determining that a set of gesture prediction parameters is not received at a time associated with a time at which the virtual content is to be generated; determining an updated prediction time based on a current time and a previous gesture prediction time based on determining that the set of gesture prediction parameters was not received at the time associated with the time when the virtual content is to be generated; and The pose of the second device is predicted based on a previously received set of pose prediction parameters and the updated prediction time.

17. The first device of claim 1, wherein the at least one processor is configured to: The one or more gestures of the first device are further predicted based on a common clock reference.

18. The first device of claim 1, wherein the at least one processor is configured to: The one or more poses of the first device are predicted further based on a predicted time when at least one frame will be rendered.

19. A first device for processing virtual data in a distributed extended reality system, the first device comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: sending one or more gesture prediction parameter sets to a second device associated with the distributed extended reality system, wherein each respective gesture prediction parameter set from the one or more gesture prediction parameter sets comprises a respective prediction coefficient and a respective prediction time associated with respective raw gesture data; receiving, from the second device, virtual content based on a gesture of the one or more predicted gestures of the first device, the one or more predicted gestures being predicted based on the one or more gesture prediction parameter sets sent to the second device; and One or more virtual objects are displayed based at least in part on the received virtual content.

20. The first device of claim 19, wherein each respective set of gesture prediction parameters from the one or more sets of gesture prediction parameters further comprises the respective raw gesture data.

21. The first device of claim 20, wherein the respective raw pose data included in each respective set of pose prediction parameters is based on respective inertial measurement unit measurements associated with at least one sensor of the first device.

22. A first device according to claim 19, wherein the one or more predicted gestures of the first device include at least one future gesture of the first device at a future time relative to a prediction time included in a gesture prediction parameter set from the one or more gesture prediction parameter sets.

23. The first device of claim 22, wherein the at least one future pose comprises a relative head pose rotation at the future time and a relative head pose translation at the future time.

24. The first device of claim 22, wherein the at least one processor is configured to: determining a relative display position of the one or more virtual objects based at least in part on the at least one future gesture of a user of the first device; and The one or more virtual objects are displayed based at least in part on the relative display positioning.

25. A method for processing virtual data at a first device in a distributed extended reality system, comprising: sending one or more gesture prediction parameter sets to a second device associated with the distributed extended reality system, wherein each respective gesture prediction parameter set from the one or more gesture prediction parameter sets comprises a respective prediction coefficient and a respective prediction time associated with respective raw gesture data; receiving, from the second device, virtual content based on a gesture of the one or more predicted gestures of the first device, the one or more predicted gestures being predicted based on the one or more gesture prediction parameter sets sent to the second device; and One or more virtual objects are displayed based at least in part on the received virtual content.

26. The method of claim 25, wherein each respective pose prediction parameter set from the one or more pose prediction parameter sets further comprises the respective raw pose data.

27. The method of claim 26, wherein the respective raw pose data included in each respective set of pose prediction parameters is based on respective inertial measurement unit measurements associated with at least one sensor of the first device.

28. The method of claim 25, wherein the one or more predicted gestures of the first device include at least one future gesture of the first device at a future time relative to a prediction time included in a gesture prediction parameter set from the one or more gesture prediction parameter sets.

29. The method of claim 28, wherein the at least one future pose comprises a relative head pose rotation at the future time and a relative head pose translation at the future time.

30. The method of claim 28, further comprising: determining a relative display position of the one or more virtual objects based at least in part on the at least one future gesture of a user of the first device; as well as The one or more virtual objects are displayed based at least in part on the relative display positioning.

Images