Server time adjustment in an extended reality (XR) environment

Abstract

A simulation time controller (STC) (130) executing on a computing device, such as a network node (100) or a client device (200), for example, controls a simulation engine (SE) (130) in an eXtended Reality environment by setting / adjusting an internal reference timer at the SE according to an optimized simulation time. The STC receives first and second sets of parameters from first and second predictor functions (120, 220), respectively. The first and second sets of parameters include respective prediction reference information for each predictor function. Additionally, the first predictor function also sends a timestamp TC indicating when an image of an actual object was captured by a camera, while the second predictor function sends a timestamp TD indicating when an image of a virtual object associated with the actual object is predicted to be displayed at the client device. Based on the received parameter sets, the STC generates a simulation timestamp TS and uses that timestamp to set / adjust the value of the internal reference timer at the SE.

Description

[0001] SERVER TIME ADJUSTMENT IN AN EXTENDED REALITY (XR) ENVIRONMENT

[0002] TECHNICAL FIELD

[0003] This application relates generally to extended Reality (XR) applications, and more particularly to controlling time adjustments in a simulation engine operating in an XR environment.

[0004] BACKGROUND

[0005] The term “extended Reality” (XR) is an “umbrella” term that refers to Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) technologies. Generally, these technologies provide users with an immersive experience by combining the physical world with a “digital world” with which the users interact.

[0006] With XR technologies, detecting the real environment is a key task. For example, existing applications provide users with the ability to interact with real and virtual objects in realtime. To accomplish this function, applications require fast and accurate knowledge about the current pose of a real moving object (i.e., the position and orientation of the real object). However, object detection takes time. Therefore, pose information about a detected real object is already outdated by time it arrives to update the Virtual World state in a simulation engine (SE). To mitigate the negative effects of this delay, conventional XR applications predict the detected pose of the object to the current (or future display) time.

[0007] SUMMARY

[0008] The present disclosure provides a computing device, such as a network node or client device, for example, specially configured with a latency sensitive end-to-end (e2e) XR application that detects objects in a real environment, as well as user input. As will be described in more detail below, the XR application of the present disclosure mitigates the latency that negatively affects conventional XR applications by predicting the pose of an object both before and after processing at a simulation engine. Particularly, the present embodiments provide a simulation time controller component that dynamically sets an internal reference timer at the simulation engine to an optimized value, thereby enhancing the quality of the simulation engine output and providing the user with a better user experience.

[0009] Accordingly, in a first aspect, the present disclosure provides a method for controlling a simulation engine in an extended Reality (XR) environment. The method in this aspect is implemented at a computing device in a communications network and comprises receiving, from a first predictor function, a first set of parameters comprising prediction reference information of the first predictor function and a timestamp Tcthat indicates when an image of an actual object was captured, receiving, from a second predictor function, a second set of parameters comprising prediction reference information of the second predictor function and a timestamp TDthat indicates when an image of a virtual object associated with the actual object is predicted to be displayed at a client device, generating a simulation timestamp Tsbased on the first and second set of parameters, and setting a time reference at the simulation engine according to the simulation timestamp Ts.

[0010] In a second aspect, the present disclosure provides a computing device configured to control a simulation engine in an extended Reality (XR) environment. In this aspect, the computing device is configured to receive, from a first predictor function, a first set of parameters comprising prediction reference information of the first predictor function and a timestamp Tcthat indicates when an image of an actual object was captured, receive, from a second predictor function, a second set of parameters comprising prediction reference information of the second predictor function and a timestamp TDthat indicates when an image of a virtual object associated with the actual object is predicted to be displayed at a client device, generate a simulation timestamp Tsbased on the first and second set of parameters, and set a time reference at the simulation engine according to the simulation timestamp Ts.

[0011] In a third aspect, the present disclosure provides a computing device configured to control a simulation engine in an extended Reality (XR) environment. In this aspect, the computing device comprises communication circuitry configured to communicate with a client device and processing circuitry operatively connected to the communication circuitry. In this aspect, the processing circuitry is configured to receive, from a first predictor function, a first set of parameters comprising prediction reference information of the first predictor function and a timestamp Tcthat indicates when an image of an actual object was captured, receive, from a second predictor function, a second set of parameters comprising prediction reference information of the second predictor function and a timestamp TDthat indicates when an image of a virtual object associated with the actual object is predicted to be displayed at a client device, generate a simulation timestamp Tsbased on the first and second set of parameters, and set a time reference at the simulation engine according to the simulation timestamp Ts.

[0012] In a fourth aspect, the present disclosure provides a non-transitory computer-readable storage medium comprising a computer program stored thereon. In this aspect, the computer program comprises executable instructions that, when executed by processing circuitry in a computing device, causes the computing device to receive, from a first predictor function, a first set of parameters comprising prediction reference information of the first predictor function and a timestamp Tcthat indicates when an image of an actual object was captured, receive, from a second predictor function, a second set of parameters comprising prediction reference information of the second predictor function and a timestamp TDthat indicates when an image of a virtual object associated with the actual object is predicted to be displayed at a client device, generate a simulation timestamp Tsbased on the first and second set of parameters, and set a time reference at the simulation engine according to the simulation timestamp Ts.

[0013] BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a functional block diagram illustrating a communications system configured according to one embodiment of the present disclosure. Figure 2 is a functional block diagram illustrating an exemplary XR application implementation.

[0015] Figure 3 is a functional block diagram illustrating an exemplary XR application implementation configured according to embodiments of the present disclosure.

[0016] Figure 4 is a functional block diagram illustrating the flow of information between a Simulation Time Controller (STC) and other components of an XR system according to embodiments of the present disclosure.

[0017] Figure 5 is a flow diagram illustrating a method for controlling a time adjustment at an STC according to embodiments of the present disclosure.

[0018] Figure 6 is a flow diagram illustrating a method for determining a time reference adjustment value for controlling a time adjustment at a SE according to embodiments of the present disclosure.

[0019] Figure 7 is a functional block diagram illustrating some of the components of a network node configured to control a time adjustment at a SE according to embodiments of the present disclosure.

[0020] Figure 8 is a functional block diagram illustrating some of the components of a client device configured to control a time adjustment at a SE according to embodiments of the present disclosure.

[0021] DETAILED DESCRIPTION

[0022] As stated previously, XR applications require fast, accurate knowledge about the position and orientation of an object it detects in the real world (i.e., the “pose” of the object). This is because simulation engines (SEs) use this information to simulate the object’s position and orientation as it moves through a virtual world for a user. However, detecting real objects in the real world takes time. As such, pose information about a detected real object is already outdated by time it is used to update a virtual world state maintained by the SE. Therefore, to mitigate the negative effects of this delay, conventional XR applications simply predict the detected pose of the object out to a current (or future display) time.

[0023] Most, if not all SEs, have an internal reference timer that controls simulation processing. The internal reference timer, also referred to as a simulation timer Ts, is not a hardware-based timer or clock, but rather, is a software timer internal to the SE. How the internal reference timer relates to the time at which a camera captured an image of the object (referred to herein as the camera time Tc) and the time at which the object in the captured image is displayed at a client device (referred to herein as the display time TD) has significant effects on the accuracy and quality of the data (e.g., image and / or video) being displayed at the client device. Additionally, prediction capabilities typically change over time and / or with the various components of the XR application, thereby significantly altering the relationship between Ts, Tc, and TD. This can, and usually does, negatively affect the end user experience. Accordingly, embodiments of the present disclosure address these and other issues by configuring a computing device, such as a network node or a client device, with a latency sensitive end-to-end (e2e) XR application that detects both objects in a real environment and user input. As will be described in more detail below, the XR application of the present disclosure mitigates the latency that negatively affects conventional XR applications by predicting the pose of an object both before and after processing at a simulation engine. Particularly, the present embodiments provide the computing device with a simulation time controller component that dynamically sets the internal reference timer at the simulation engine to an optimized value of Ts, thereby enhancing the quality of the simulation engine output and providing the user with a better user experience.

[0024] In more detail, the present embodiments provide a simulation plugin module operatively connected to the SE (e.g., via an Application Programming Interface (API), for example), and a simulation time controller (STC) that communicates with the SE plugin module. The STC is further communicatively connected to first and second object predictor functions. The first object predictor function is configured to predict the pose of a detected real object prior to SE processing, and the second object predictor function is configured to predict the pose of the real object and / or a virtual object associated with the real object after SE processing. To accomplish its function, the STC of the present disclosure obtains (e.g., receives) first and second sets of one or more parameters from the first and second object predictor functions, respectively. Then, based (at least in part) on the first and second parameter sets, the STC of the present disclosure generates (e.g., calculates) an optimal time value for Ts, and dynamically sets / adjusts the internal reference timer of the SE according to Ts.

[0025] In some embodiments, calculating the optimal time value for Ts and setting / adjusting the internal time reference of the SE according to the calculated Tsvalue is performed on a perframe granularity. Additionally, dynamically calculating Tsand sending it to the SE plugin module allows the STC to control the simulation functions of the SE. For example, providing the optimized Tsto the SE plugin module of the present embodiments causes the SE plugin module to interact with the SE simulation logic to delay or “speed up” execution of the simulation logic, as well as to determine the arrival time of other useful information and data, such as the user input.

[0026] The STC of the present disclosure provides benefits and advantages for XR applications that conventionally configured XR applications cannot or do not provide. For example, according to the present embodiments, the object prediction functions are split to occur before processing by the SE, and after processing by the SE (e.g., at a client HMD). Additionally, the distribution of the prediction functionality between the first and second predictor functions (i.e., the amount of prediction processing each of the first and second prediction functions is to perform) is a tradeoff governed by different aspects that affect the end-user experience. Thus, the present embodiments dynamically calculate and set, e.g., on a per-frame granularity, the internal reference timer of the SE to an optimal simulation time Ts. Doing so enables the SE of the present disclosure to better react to changes in network delay (e.g., that affect communications between the HMD and the SE) and in quality (e.g., changes in the quality of the network, optical quality, etc.). It also enhances the capabilities of the prediction function at the client device (e.g., the HMD) to perform environment-level object detection and increases the sensitivity to prediction error in various events and / or situations (e.g., game events / simulations).

[0027] For example, consider a situation in which a detected “real” object moves slowly in a straight line. Because of this lack of speed / movement, the first prediction function (e.g., at the network node) is able to detect and predict the pose of the object with a very high accuracy, and as a result, reports high quality pose data to the STC. The STC, in turn, generates a value for Tsto be closer to the value of TD, and then sets / adjusts the internal reference timer at the SE according to the generated value of Ts. Due to the high quality of the pose data, the setting / adjustment causes the SE to limit the post-processing of an image, thereby achieving high image quality at the client device.

[0028] However, when the detected “real” object begins to move faster and / or unpredictably, or the input to an object detector function is interrupted (e.g., the camera capturing the images is covered up), the first prediction function reports reduced quality pose data to the STC. This then causes the SE to generate the value of Tsto be closer to that of Tc. The STC then sets / adjusts the internal reference timer at the SE according to the generated Ts. This configures the SE to be more aligned with the detected environment at or near the time the image of the real object was captured (i.e., Tc), and to react to real world events rather than what could most likely be false predictions generated by the first prediction function. In the end, though, the image quality at the client device is of a higher quality.

[0029] As another example, consider prediction quality changes at the client device (e.g., the HMD). In one situation, the HMD is running optimally (e.g., the battery level is high, low temperature, etc.) and can run intense computation tasks. Therefore, the predictor function at the HMD reports high quality prediction data to the STC. As a result, the STC generates Tsto be closer to the value of Tc, and then sets / adjusts the value of the internal internal reference timer at the SE according to Ts. Then, when the power at the HMD gets low (or the HMD overheats, for example), the SE will have to reduce the computational load at the HMD by reducing the number or amount of intense tasks that the HMD will have to process. Such a reduction, however, results in a degraded prediction quality reported to the STC by the HMD prediction function, which in turn, causes the STC to generate the value of Tsto be closer to the value of TD. The STC then sets / adjusts the internal reference timer at the SE according to Ts, thereby increasing the computational load on the predictor function at the computing device and reducing the computational load on the prediction function at the HMD.

[0030] Turning now to the drawings, Figure 1 is a functional block diagram illustrating a communications network 10 configured according to one embodiment of the present disclosure. As seen in Figure 1 , network 10 comprises an access network 12 communicatively connecting a client device 200 (e.g., a HMD) with a network node 100 (e.g., a server node) disposed in a cloud network 14. In some embodiments, a computing device 20 may be disposed between client device 200 and server node 100 and is configured to perform at least some of the processing functions of client device 200.

[0031] The access network 12 may be any type of communications network (e.g., Wireless Fidelity (WiFi), ETHERNET, Wireless Local Area Network (WLAN), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), etc.), and functions to connect subscriber devices, such as client device 200, to one or more service provider nodes, such as network node 100. The cloud network 14 provides such subscriber devices with “on-demand” availability of computer resources (e.g., memory, data storage, processing power, etc.) without requiring the user to directly, actively manage those resources. According to embodiments of the present disclosure, such resources include, but are not limited to, one or more XR applications being executed on network node 100. The XR applications, which are described in more detail below, may comprise, for example, those used in connection with gaming applications and / or simulation applications.

[0032] It should be noted here that the embodiment of Figure 1 , as well as the following embodiments, illustrate a STC and a SE as being co-located on network node 100. However, this is merely illustrative, and the present embodiments are not so limited. As will be discussed in more detail later, the STC and the SE may be co-located on client device 200 (e.g., on the user’s HMD, for example) or on any computing device, such as computing device 20, the user’s laptop computer, notebook computer, desktop computer, local server computer, mobile communications device - a User Equipment (UE) device - (e.g., a SMARTPHONE), tablet computer, and the like. Additionally, as with the network node 100, a computing device in these embodiments may, for example, comprise dedicated hardware circuitry, such as one or more graphics processing units (GPU), and be capable of executing games and other software programs associated with an XR environment.

[0033] Figure 2 illustrates some key functional components of an example XR application implementation, which comprises the client device 200 and the network node (e.g., server node) 100. As seen in Figure 2, sensors 210 (e.g., a camera and / or an Inertial Measurement Unit (IMU)) at client device 200 sends data (e.g., 2D / 3D images, camera pose data, and timestamp Tcindicating when sensor 210 captured the image of the object and / or detected user movement) to an object detector 110 at network node 100. The object detector 110 detects the pose of the real object from the received sensor data and sends the detected object pose information (including timestamp Tc) to an object predictor 120 at network node 100. The predictor 120 then predicts the pose of the object out to some future timestamp (i.e., as compared to the timestamp Tc) and sends the predicted object pose (including timestamp Tc) to the SE 130. The SE 130 may comprise, for example, a game engine such as UNITY. In this case, the SE 130 is responsible for determining the pose data for a virtual object associated with a real object based on the pose data provided by the predictor 120 and the laws of physics. SE 130 performs the XR application logic in addition to the simulation logic and may implement some client-specific steps as well. For example, SE 130 may implement the processing required to render one or more images for display at client device 200. Regardless, though, SE 100 sends client-specific updates (e.g., game data, pose data, rendered image(s), etc.) to a predictor function 220 at client device 200. Upon receipt, predictor 220 predicts the pose data out for the object to an estimated display time TDand displays the predicted data on display 230.

[0034] Figure 3 is a functional block diagram illustrating the key components of an exemplary XR application implementation configured according to embodiments of the present disclosure. The key components in Figure 3 are the same as those illustrated in Figure 2 (i.e., client device 200 and network node 100). Further, each of these components comprise the same functions / modules and operate in the same manner as those previously described in Figure 2 (i.e., the object detector 110, predictor 120, SE 130, sensors 210, predictor 220, display 230). However, in accordance with the present embodiments, SE 130 is configured to further comprise a simulation plugin module 140 (also referred to herein as “plugin 140”). Additionally, the network node 100 is configured to further comprise the simulation time controller (STC) 150. The addition of plugin 140 and STC 150, and their ability to communicate with each other, configures network node 100 to dynamically calculate and set / adjust the internal reference timer of SE 130 according to the generated optimal simulation time Tsat a per-frame granularity. As stated above, this enables the SE 130 to better react to changes in network delay and quality. It also enhances the capabilities of the predictor 220 at client device 200 to perform environmentlevel object detection and increases the sensitivity to prediction error in a variety of different events and / or situations.

[0035] In more detail, this embodiment assumes, for the sake of illustration, that all component functions (i.e., the object detector 110, predictor 120, SE 130, sensors 210, predictor 220, display 230) are time synced. As previously described, Tcindicates the timestamp when sensor 210 (e.g., the camera at client device 200) captured an image [i] of a real object. Thus, Tc[i] is the timestamp to which the information in picture [i] belongs. Similarly, Ts[i] is the timestamp when that same picture [i] (or the object pose information derived from picture [i]) is processed by SE 130, and TD[i] is the timestamp when the corresponding content was displayed at display 230. Therefore, if FPSC, FPSs, FPSDdenotes the framerates (i.e., frames per second) of sensor 210, the simulation engine 130, and display 230, respectively, then in ideal circumstances:

[0036] Tx[i+1] = Tx[i] + 1 / FPSx where x is C, S, or D. The data flow between the functions in this embodiment is the same as that previously described in Figure 2. However, as seen in the embodiment of Figure 3, data also flows between STC 150 and each of predictor 120, the plugin 140, and predictor 220.

[0037] Particularly, Figure 4 is a functional block diagram illustrating the flow of information between STC 150 and other functions of an XR application according to embodiments of the present disclosure. As best seen in Figure 4, predictors 120 and 220 each provide STC 150 with prediction reference information. This information comprises, in at least one embodiment, data that identifies the respective prediction capabilities Ci, C2of predictors 120, 220, the respective prediction quality Qi, Q2of predictors 120, 220, and a prediction time period TPfor SE 130. The prediction capabilities information Ci, C2indicates the capabilities and / or constraints of predictor 120, 220. For example, consider a situation where predictor 120 is unable to predict the pose of a real object in image [i] more than 200 milliseconds into the future. In these cases, this time constraint is communicated to STC 150 to ensure that STC 150 will not determine or calculate the value of Tsto be more than 200 ms ahead of the value of Tc. The prediction quality Qi, Q2indicates the output quality of the predictor 120, 220, respectively, and the prediction time period TPindicates a time period over which SE 130 is to predict the behavior of the real object captured in image [i] at time Tc[i] .

[0038] Additionally, predictors 120, 220 provide STC 150 with timestamps Tcand TD, respectively. According to the present disclosure, an exact time for TDmay not be known. However, in at least one embodiment, client device 200 is configured to store previous timestamps TDand provide them to STC 150. Additionally, or alternatively, client device 200 will know when the image [i] will be displayed, and thus, will know a more accurate value for TD. In any case, STC 150 can, according to the present embodiments, derive timestamp TDfrom the earlier timestamps TDor be provided with the more accurate value for timestamp TDby client device 200.

[0039] Further, plugin 140 at SE 130 provides STC 150 with data detailing an accuracy requirement A for the simulation logic being executed by SE 130, as well as user input information (e.g., action time(s) indicating when the user performed a given action and a simulator receive time). In at least one embodiment, plugin 140 also optionally provides STC 150 with a desired ATS. According to the present disclosure, the desired ATSis defined by the simulation logic of SE 130 and indicates a valid timestamp range for the simulation timestamp Ts. Upon receipt, STC 150 uses the desired ATSto ensure that the generated Tsfalls within the valid timestamp range specified by the plugin 140. If the generated Tsdoes not fall within the valid timestamp range, STC 150 can adjust the generated Tsaccordingly.

[0040] The STC 150 of the present disclosure also provides predictors 120, 220, and plugin 140 with the generated Ts. Upon receipt, predictors 120, 220 each utilize Tsin their respective prediction operations, while plugin 140 sets / adjusts the internal reference timer of SE 130 (e.g., via the API) according to the value of Ts. As stated above, STC 150 is configured according to the present disclosure to determine an optimal value for Tsbased on the parameters provided by predictors 120, 220 and by SE 130 via plugin 140. However, there is a trade-off in moving Tscloser to the value of Tcas opposed to moving Tscloser to the value of TD. As an example, let tpredi denote the amount of time needed by predictor 120 to predict the pose of the object in the image before providing the resultant pose data to SE 130. Similarly, let tpred2 denote the amount of time predictor 220 needs to predict the pose of the object or the object image after receiving the output from SE 130. It follows, then, that: tpredi = Ts — Tc = TD — Tc — tpred2j tpred2 = TD — Ts = TD — Tc — tpredi I tpredi + tpred2—TD - Tc, Tc + tpredi = Ts; and Ts + tpred2 = TD.

[0041] Determining the optimal value for Tswill depend on whether the value of Tsis closer to the value of Tcor the value of TD. Each will have its own advantages. For instance:

[0042] Example 1 : In this example, Tsis determined to be as close to TDas possible (i.e., tpred2 —> 0). This indicates little (or no) user input (e.g., head movement, etc.) for predictor 220 to process, and means that pose predictions were performed (entirely or mostly) on the real object by predictor 120. In these cases, more processing work is performed prior to simulation; however, the user will perceive a better picture quality, as the rendered image is higher in quality than the if it were to be transformed. Therefore, in an ideal case:

[0043] Ts= TD;

[0044] However, in all cases, the determined Tsmust not overshoot TD. Thus, in general:

[0045] Ts TD

[0046] Example 2: In another example, the present embodiments determine Tsto be closer to Tc(i.e., tpredi —> 0). This lets all (or mostly all) information arrive at SE 130 before rendering the image(s) or pose data at SE 130. Although this would advantageously practically eliminate prediction processing at predictor 120 prior to SE 130, it also means increased prediction processing at predictor 220. It also means relying on predictor 220 to accurately perform its prediction functions in different scenarios relative to one or more unpredictable events (e.g., events that occur arbitrarily, and thus, cannot be predicted by the XR application). Such unpredictable events include, but are not limited to, user actions (e.g., head movement, eye movement, etc.) and / or changes in the movement of the real object.

[0047] Thus, in accordance with the present disclosure, STC 150 is configured to balance the trade-off that exists between determining the value of Tsbetween those of Tcand TD. As an example, consider a situation assuming the following inputs:

[0048] • TD , Tc;

[0049] • a current prediction quality Q available for predictors 120, 220: Qi (tpredi) , Q2(tpred2) in which [0<Qx<1], and wherein: when Qx=O, the prediction does not work; and when Qx=1 , the prediction is almost perfect;

[0050] • a current simulation logic accuracy requirement A: in which [0<A<1], and wherein: when A=0, it corresponds to Example 1 above (i.e., where user or environment input is not as important, such as in a scene where the user and / or object is moving through empty space); and when A=1 , it corresponds to Example 2 above (i.e., where user and / or environment input is important, such as in a scene where a user is shooting at an avatar).

[0051] Formalizing the trade-off for each frame based on the input provides: tpredl + tpred2—TD_TO

[0052] Then, the value of tpredi is weighted against the value of tpred2 according to— . Such weighting

[0053] 22reflects the difference in prediction qualities before and after processing at SE 130. That is:

[0054] Ql tpredi

[0055] Q 2 tpred 2

[0056] The value of tpredi is then weighted against the value of tpred2 according to the value of 1-A. This allows for the consideration of the requisite accuracy of the SE 130:

[0057] (1— x4) tpred\

[0058] A tpred 2

[0059] Then, if possible, the value of tpred2 is minimized, thereby reducing image transformations (e.g., warp), which can advantageously provide a better rendered image quality.

[0060] In light of the above, one example calculation that can be performed by STC 150 to generate an optimal Tsbased on the considerations above is:

[0061] It should be noted here that the above approach is generic, and as such, the equation above used to determine Tscan be applied on a framerate of the components. Therefore, when new input data arrives at STC 150, it calculates and configures a new Ts. Alternatively, STC 150 can apply the above equation with hysteresis to relax instant reaction on temporal effects.

[0062] Additionally, the present embodiments do not determine timestamp Tsfor previous time periods (i.e., the present embodiments do not move Tsbackwards in time). This allows STC 150, and thus, SE 130, to avoid various undesirable side effects. Instead, the present embodiments calculate Tssuch that upon setting / adjusting the internal reference timer of SE 130, simulation processing by SE 130 is slowed but not stopped entirely. Similarly, the present embodiments also avoid significant jumps in Ts. To accomplish this, the present embodiments are configured to distribute the Tscalculations by determining multiple Tsacross multiple frames.

[0063] Additionally, the respective output quality Q (i.e., Qi, Q2of predictors 120, 220) may change as a function of Ts-Tcand / or TD-TS, respectively. Similarly, the respective output quality Q of predictors 120, 220, and / or the accuracy requirement A of the simulation functions being executed by SE 130, can change over time. Regardless, the STC 150 considers the effects of each of these changes and determines the value of Tsaccordingly.

[0064] It should be noted here that the previous embodiments illustrate the STC 150 as being in the cloud and co-located with SE 130 on network node 100 along with plugin 140. However, those of ordinary skill in the art should readily appreciate that this is for illustrative purposes and ease of discussion only. The present embodiments do not restrict the placement of STC 150 to being on network node 100 or co-located with SE 130. Rather, the present embodiments allow for STC 150 to be disposed on a different network node 100 so long as it can communicate with the predictors 120, 220, and the plugin 140. Additionally, or alternatively, one embodiment of the present disclosure integrates STC 150 into SE 130. Such integration may, for example, comprise integrating the coding for STC 150 into the coding of SE 130. Another embodiment provides the coding for STC 150 as a plugin module configured to be communicatively connected to SE 130.

[0065] Figure 5 is a flow diagram illustrating a method 300 for controlling a time adjustment at SE 130 according to embodiments of the present disclosure. In this embodiment, method 300 is implemented at network node 100 and comprises the STC 150 receiving, from a first predictor function (e.g., predictor 120 at network node 100), a first set of parameters (box 302). In this embodiment, the first set of parameters comprises prediction reference information of the first predictor function and a timestamp Tcthat indicates when an image of an actual object was captured. Method 300 also calls for STC 150 receiving, from a second predictor function (e.g., predictor 220 at client device 200), a second set of parameters (box 304). The second set of parameters comprises, in this embodiment, prediction reference information of the second predictor function and a timestamp TDthat indicates when an image of a virtual object associated with the actual object is predicted to be displayed at a client device. Provided with this input, STC 150 then generates a simulation timestamp Tsbased on the first and second set of parameters (box 306) and sets a time reference at SE 130 according to the simulation timestamp Ts(box 308).

[0066] Figure 6 is a flow diagram illustrating a method 310 for determining the value of Tsfor controlling a time adjustment at SE 130 according to embodiments of the present disclosure. In this embodiment, method 310 is implemented by STC 150, which as stated above, may or may not be co-located with the plugin 140 on network node 100. As seen in Figure 6, STC 150 receives, from SE 130, one or more of the accuracy requirement parameter A indicating a desired accuracy for the simulation timestamp Tsand one or more user input parameters (box 312). The user input parameters may comprise, as stated above, data regarding the user’s movements (e.g., movements of the user’s head, eyes, and / or extremities) and actions (e.g., actuating a trigger or other user control) at client device 200. Additionally, STC 150 also receives, from the simulation engine, a valid timestamp range for the simulation timestamp Ts(box 314). The STC 150 then generates the simulation timestamp Tsbased further on the accuracy requirement parameter A (box 316), and in some embodiments, sends the generated timestamp Tsto the first predictor function (box 318).

[0067] In one embodiment, the time reference comprises an internal timer of the simulation engine.

[0068] In one embodiment, the prediction reference information received from the first predictor function comprises one or more of prediction capability data indicating one or more capabilities and / or constraints of the first predictor function and prediction quality data indicating a quality of the first predictor function output.

[0069] In one embodiment, the prediction reference information received from the second predictor function comprises one or more of prediction capability data indicating one or more capabilities and / or constraints of the second predictor function and prediction quality data indicating a quality of the second predictor function output.

[0070] In one embodiment, the first predictor function provides the simulation engine with predicted pose information for the actual object in the image, and the second predictor function receives rendered image data from the simulation engine.

[0071] In one embodiment, the second predictor function receives predicted pose information for the virtual object from the simulation engine.

[0072] In one embodiment, the accuracy requirement parameter, the one or more user input parameters, and the valid timestamp range are received from a simulator engine plugin operatively connected to the simulation engine.

[0073] In one embodiment, the simulation timestamp Tsis bounded by Tcand TDsuch that:

[0074] Tc < Ts < TD .

[0075] In one embodiment, the simulation timestamp Tsis adjustable between Tcand TD.

[0076] In one embodiment, the simulation timestamp Tsis generated according to: wherein:

[0077] A is a value [0 < A < 1] representing the accuracy requirement received from the simulation engine,

[0078] Qi is a value [0 < Qi < 1] representing the quality of the first predictor function output, and Q2is a value [0 < Q2< 1] representing the quality of the second predictor function output.

[0079] In one embodiment, one or both of Qi and / or Q2vary over time.

[0080] In another embodiment, however, Qi varies relative to Ts-Tc, and Q2varies relative to TD- Ts.

[0081] In one embodiment, the accuracy requirement A varies over time.

[0082] In one embodiment, the simulation timestamp TS is generated on a per-frame basis.

[0083] In one embodiment, the time reference at the simulation engine is set according to the simulation timestamp TS on the per-frame basis.

[0084] In one embodiment, the simulation timestamp TS is generated responsive to the computing device receiving an updated first set of parameters from the first predictor function.

[0085] In one embodiment, the simulation timestamp Tsis generated responsive to the computing device receiving an updated second set of parameters from the second predictor function.

[0086] In one embodiment, the simulation timestamp Tsis generated responsive to the computing device receiving one or more of the accuracy requirement parameter, the one or more user input parameters, and the valid timestamp range.

[0087] In one embodiment, the computing device is a network node in the communications network.

[0088] In another embodiment, however, the computing device is a client device in the communications network. Such a client device may be, for example, a head mounted display (HMD).

[0089] An apparatus can perform any of the methods herein described by implementing any functional means, modules, units, or circuitry. In one embodiment, for example, the apparatuses comprise respective circuits or circuitry configured to perform the steps shown in the method figures. The circuits or circuitry in this regard may comprise circuits dedicated to performing certain functional processing and / or one or more microprocessors in conjunction with memory. For instance, the circuitry may include one or more microprocessors or microcontrollers, as well as other digital hardware, which may include Digital Signal Processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory may include program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein, in several embodiments. In embodiments that employ memory, the memory stores program code that, when executed by the one or more processors, carries out the techniques described herein.

[0090] Figure 7 is a functional block diagram illustrating some of the components of a network node 100 configured to control a time adjustment at SE 130 according to embodiments of the present disclosure. The network node 100 in this embodiment is configured to execute STC 150 and SE 130 and comprises, inter alia, communication circuitry 102, processing circuitry 104, and memory 106.

[0091] In some embodiments, the communication circuitry 102 comprises both radio frequency (RF) circuitry 102a and network interface circuitry (NIC) 120b. In other embodiments, however, the network node may comprise only NIC 102b. More particularly, the RF circuitry 102a can be located at one or more TRPs and comprises the RF components necessary for communicating with various client devices 200 over a wireless communication link. According to the present embodiments, the RF circuitry 102a may comprise, for example, a transmitter and receiver configured to operate according to the 5G standards or other wireless communication standard.

[0092] The communication circuitry 102 also comprises network interface circuitry (e.g., NIC 102b) for communication with other RAN nodes, OA&M nodes, core network nodes, and / or other nodes in external systems. The network interface circuitry 102b in this regard may, for example, comprise an Ethernet interface, optical network interface, or a wireless interface.

[0093] The processing circuitry 104 comprises one or more microprocessors, hardware, firmware, or a combination thereof that controls the overall operation of the network node 100. The processing circuitry 104 in this regard can be configured by software to perform the functionality described with respect to Figures 1-4, and one or more of the methods herein described, including methods 300 and 310 as shown in Figures 5-6, respectively.

[0094] Memory 106 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuitry 104 for operation. Memory 106 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 106 stores one or more computer programs 108a (e.g., STC 150), 108b (e.g., plugin 140) comprising executable instructions that configure the processing circuitry 104 in the network node 100 to perform one or more of the methods herein described, including the methods 300 and 310 as shown in Figures 5-6, respectively. A computer program 108a, 108b in this regard may comprise one or more code modules corresponding to the means or units described above.

[0095] In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer programs 108a and / or 108b for configuring the processing circuitry 104 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer programs 108a and / or 108b may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium. As stated above, the STC 150 and the SE 130 may be co-located on a network node disposed in the cloud, for example, or they may be co-located on another device, such as client device 200 (e.g., a HMD), a UE (smartphone) configured to send and receive data over a mobile communications network, computing device 20, or any computing device that is capable of executing simulation processing (e.g., a game engine). And that can communicate with the functions of client device 200.

[0096] For example, Figure 8 is a functional block diagram illustrating some of the components of a client device 200 configured to control a time adjustment at SE 130 according to embodiments of the present disclosure. The client device 200 in this embodiment may be, for example, a HMD worn by a user, a UE (smartphone) configured to send and receive data over a mobile communications network, or a computing device, each capable of executing SE 130 and STC 150. Regardless, though, the client device 200 in this embodiment is configured to execute STC 150 and SE 130 and comprises, inter alia, communication circuitry 202, processing circuitry 204, and memory 206.

[0097] The communication circuitry 202 in this embodiment comprises both radio frequency (RF) circuitry 202a and network interface circuitry (NIC) 202b. In other embodiments, however, client device 200 may comprise only the NIC 202b. More particularly, the RF circuitry 202a can be located at one or more TRPs and comprises the RF components necessary for communicating with network node 100 and / or other devices over a wireless communication link. According to the present embodiments, the RF circuitry 202a may comprise, for example, a transmitter and receiver configured to operate according to the 5G standards or other wireless communication standard.

[0098] The communication circuitry 202 also comprises network interface circuitry (e.g., NIC 202b) for communication with other RAN nodes, OA&M nodes, core network nodes, and / or other nodes in external systems. The network interface circuitry 202b in this regard may, for example, comprise an ETHERNET interface, optical network interface, or a wireless interface.

[0099] The processing circuitry 204 comprises one or more microprocessors, hardware, firmware, or a combination thereof that controls the overall operation of the client device 200. The processing circuitry 204 in this regard can be configured by software to perform the functionality described with respect to Figures 1-4, and one or more of the methods herein described, including methods 300 and 310 as shown in Figures 5-6, respectively.

[0100] Memory 206 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuitry 204 for operation. Memory 206 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage.

[0101] Memory 206 stores one or more computer programs 208a (e.g., STC 150), 208b (e.g., plugin 140) comprising executable instructions that configure the processing circuitry 204 in the client device 200 (or other computing device) to perform one or more of the methods herein described, including the methods 300 and 310 as shown in Figures 5-6, respectively. A computer program 208a, 208b in this regard may comprise one or more code modules corresponding to the means or units described above.

[0102] In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer programs 208a and / or 208b for configuring the processing circuitry 204 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer programs 208a and / or 208b may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0103] Those skilled in the art will also appreciate that embodiments herein further include corresponding computer programs. A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

[0104] Embodiments of the present disclosure further include a carrier containing such a computer program 108a, 108b, 208a, and / or208b. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0105] In this regard, the embodiments described herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform as described above.

[0106] Embodiments further include a computer program product comprising program code portions for performing the method of any of the embodiments herein when the computer program product is executed by a computing device, such as network node 100, client device 200, or computing device 20. This computer program product may be stored on a computer readable recording medium.

[0107] Embodiments further include a user equipment (UE) configured to perform the functionality described with respect to Figures 1-4, and one or more of the methods herein described, including methods 300 and 310 as shown in Figures 5-6, respectively.

[0108] The present embodiments may, of course, be carried out in other ways than those specifically set forth herein without departing from characteristics described herein. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, and all changes coming within the meaning and equivalency range of the appended claims are intended to be embraced therein.

Claims

CLAIMS1. A method (300) for controlling a simulation engine (130) in an extended Reality (XR) environment, the method implemented at a computing device in a communications network (10) and comprising: receiving (302), from a first predictor function (120), a first set of parameters comprising prediction reference information of the first predictor function and a timestamp Tcthat indicates when an image of an actual object was captured; receiving (304), from a second predictor function (220), a second set of parameters comprising prediction reference information of the second predictor function and a timestamp TDthat indicates when an image of a virtual object associated with the actual object is predicted to be displayed at a client device (200); generating (306) a simulation timestamp Tsbased on the first and second set of parameters; and setting (308) a time reference at the simulation engine according to the simulation timestamp Ts.

2. The method of claim 1 , wherein the time reference comprises an internal timer of the simulation engine.

3. The method of claims 1-2, wherein the prediction reference information received from the first predictor function comprises one or more of: prediction capability data indicating one or more capabilities and / or constraints of the first predictor function; and prediction quality data indicating a quality of the first predictor function output.

4. The method of claims 1-3, wherein the prediction reference information received from the second predictor function comprises one or more of: prediction capability data indicating one or more capabilities and / or constraints of the second predictor function; and prediction quality data indicating a quality of the second predictor function output.

5. The method of any of claims 1-4, wherein the first predictor function provides the simulation engine with predicted pose information for the actual object in the image, and wherein the second predictor function receives rendered image data from the simulation engine.

6. The method of any of claims 1-5, wherein the second predictor function receives predicted pose information for the virtual object from the simulation engine.

7. The method of any of claims 1-6, further comprising receiving (312), from the simulation engine, one or more of: an accuracy requirement parameter indicating a desired accuracy for the simulation timestamp Ts; and one or more user input parameters.

8. The method of claim 7, further comprising generating (316) the simulation timestamp Tsbased further on the accuracy requirement parameter.

9. The method of any of claims 7-8, further comprising receiving (314), from the simulation engine a valid timestamp range for the simulation timestamp Ts.

10. The method of any of claims 1-9, wherein the accuracy requirement parameter, the one or more user input parameters, and the valid timestamp range are received from a simulator engine plugin (140) operatively connected to the simulation engine.

11. The method of any of claims 1-10, further comprising sending (318) the simulation timestamp Tsto the first predictor function.

12. The method of any of claims 1-11 , wherein the simulation timestamp Tsis bounded by Tcand TDsuch that:Tc < Ts < TD .

13. The method of claim 12, wherein the simulation timestamp Tsis adjustable between Tcand TD.

14. The method of any of claims 1-13, wherein the simulation timestamp Tsis generated according to:wherein:A is a value [0 < A < 1] representing the accuracy requirement received from the simulation engine;Qi is a value [0 < Qi < 1] representing the quality of the first predictor function output; and Q2is a value [0 < Q2< 1] representing the quality of the second predictor function output.

15. The method of claim 14, wherein one or both of Qi and / or Q2vary over time.

16. The method of any of claims 14-15, wherein Qi varies relative to Ts-Tc, and wherein Q2varies relative to TD-Ts.

17. The method of any of claims 14-16, wherein the accuracy requirement A varies over time.

18. The method of any of the preceding claims, wherein the simulation timestamp Tsis generated on a per-frame basis.

19. The method of claim 18, wherein the time reference at the simulation engine is set according to the simulation timestamp Tson the per-frame basis.

20. The method of any of the preceding claims, wherein the simulation timestamp Tsis generated responsive to the computing device receiving an updated first set of parameters from the first predictor function.

21. The method of any of the preceding claims, wherein the simulation timestamp Tsis generated responsive to the computing device receiving an updated second set of parameters from the second predictor function.

22. The method of any of the preceding claims, wherein the simulation timestamp Tsis generated responsive to the computing device receiving one or more of: the accuracy requirement parameter; the one or more user input parameters; and the valid timestamp range.

23. The method of any of the preceding claims, wherein the computing device is a network node (100) in the communications network.

24. The method of any of the preceding claims wherein the computing device is a client device (200) in the communications network.

25. The method of claim 24, wherein the client device is a head mounted display (HMD).

26. A computing device configured to control a simulation engine (130) in an extended Reality (XR) environment, the computing device being configured to: receive (302), from a first predictor function (120), a first set of parameters comprising prediction reference information of the first predictor function and a timestamp Tcthat indicates when an image of an actual object was captured; receive (304), from a second predictor function (220), a second set of parameters comprising prediction reference information of the second predictor function and a timestamp TDthat indicates when an image of a virtual object associated with the actual object is predicted to be displayed at a client device (200); generate (306) a simulation timestamp Tsbased on the first and second set of parameters; and set (308) a time reference at the simulation engine according to the simulation timestamp Ts.

27. The computing device of claim 26, wherein the computing device is further configured to perform the method according to any one of claims 2-25.

28. A computing device configured to control a simulation engine (130) in an extended Reality (XR) environment, the computing device comprising: communication circuitry (102) configured to communicate with a client device (200); and processing circuitry (104) operatively connected to the communication circuitry and configured to: receive (302), from a first predictor function (120), a first set of parameters comprising prediction reference information of the first predictor function and a timestamp Tcthat indicates when an image of an actual object was captured; receive (304), from a second predictor function (220), a second set of parameters comprising prediction reference information of the second predictor function and a timestamp TDthat indicates when an image of a virtual object associated with the actual object is predicted to be displayed at a client device (200); generate (306) a simulation timestamp Tsbased on the first and second set of parameters; and set (308) a time reference at the simulation engine according to the simulation timestamp Ts.

29. The computing device of claim 28, wherein the processing circuitry is further configured to perform the method according to any one of claims 2-25.

30. A computer program (108a, 108b, 208a, 208b) comprising instructions that, when executed on processing circuitry (104, 204) of a computing device (100, 200), cause the computing device to perform the method according to any of claims 1-25.31 . A carrier containing the computer program of claim 30, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

32. A non-transitory computer-readable storage medium (106) comprising a computer program (108a, 108b, 208a, 208b) stored thereon, the computer program comprising executable instructions that, when executed by processing circuitry (104, 204) in a computing device (100, 200), causes the computing device to perform the method of any one of claims 1-25.

33. A User Equipment (UE) configured to control a simulation engine (130) in an extended Reality (XR) environment, the UE being configured to: receive (302), from a first predictor function (120), a first set of parameters comprising prediction reference information of the first predictor function and a timestamp Tcthat indicates when an image of an actual object was captured; receive (304), from a second predictor function (220), a second set of parameters comprising prediction reference information of the second predictor function and a timestamp TDthat indicates when an image of a virtual object associated with the actual object is predicted to be displayed; generate (306) a simulation timestamp Tsbased on the first and second set of parameters; and set (308) a time reference at the simulation engine according to the simulation timestamp Ts.

34. The UE of claim 28, being further configured to perform the method according to any one of claims 2-25.

Images