Delayed decisions in augmented reality / mixed reality applications using object pose prediction

Abstract

A network node (200) processes Augmented Reality (AR) / Mixed Reality (MR) data for display at a user device, such as a Head Mounted display (HMD). The network node executes a simulation engine that receives (92) and utilizes (94, 96, 98, 100) both the delayed but accurate pose data generated an object detection function, and the more current but less accurate predicted pose data generated by an object prediction function. Additionally, the simulation engine also receives and utilizes accuracy information indicating the accuracy of the pose data generated by the object detection function, and the predicted pose data generated by the object prediction function. Based on this accuracy data, the simulation engine makes processing decisions for the AR / MR data.

Description

[0001] DELAYED DECISIONS IN AUGMENTED REALITY / MIXED REALITY APPLICATIONS USING OBJECT POSE PREDICTION

[0002] TECHNICAL FIELD

[0003] This application relates generally to the processing of Augmented Reality (AR) I Mixed Reality (MR) data, and more particularly to the processing of AR / MR data based on quality information.

[0004] BACKGROUND

[0005] Detecting a real object in a real environment is a key task in processing Augmented Reality (AR) I Mixed Reality (MR) data. Such detection helps facilitate real-time interactions between a user wearing a Head Mounted Display (HMD), for example, and a real and / or virtual object. Typically, to facilitate the interactions, AR / MR applications require fast, accurate knowledge about the current pose (i.e., the position and orientation) of a moving real object, such as a ball, for example. This is because the virtual “world state” decisions made by a simulation engine, such as determining whether the ball fell into a bucket or crossed a goal line, for example, are affected by the real-world state of the real object (e.g., the movement of the ball).

[0006] As is known in the art, the “world state” is part of a simulation engine, sometimes referred to as a game engine. Examples of such engines include, but are not limited to, 3D simulation engines such as UNITY or UNREAL. In these engines, the world state is a 3D model of the real (or game) world containing the representation of all objects in the real world / game world. For example, with car racing games, the world state contains virtual objects representing the real-world cars, roads, obstacles, the background environment, and the like. With AR / MR applications, however, the world state contains both the virtual objects representing the detected real-world objects, in addition to “fully virtual” objects representing objects that do not exist in the real world or are not detected.

[0007] For example, consider an AR / MR soccer game. The world state in this example could contain a virtual soccer ball object that corresponds to the real-world soccer ball. The real-world soccer ball is, of course, detected, while the virtual soccer ball is manipulated according to a predicted position and orientation. For instance, the engine may manipulate the virtual soccer ball object to follow the same trajectory as the detected real-world soccer ball. In addition, however, the world state may also contain one or more “fully virtual” objects representing things that may be in the real world but are not detected. Such fully virtual objects include objects representing things like the goal line on the field (i.e., a virtual goal line object) and a scoreboard (i.e., a virtual scoreboard object reflecting the current score).

[0008] Unfortunately, although it is a well-known process in the realm of AR / MR processing, the detection of a real-world object takes time. Thus, data representing the position and orientation of a detected real-world object (i.e., the pose data) is already outdated when received at the simulation engine (e.g., a game engine such as UNITY) for processing. To mitigate the effects of such delay, however, many AR / MR applications will predict the future position and orientation of the real-world object based on the current pose data and provide that prediction to the simulation engine.

[0009] SUMMARY

[0010] The present disclosure provides a method and apparatus for processing Augmented Reality (AR) I Mixed Reality (MR) data for display at a user device, such as a Head Mounted display (HMD). In particular, a simulation engine receives and utilizes the delayed but accurate pose data generated by an object detection node, and the more current but less accurate predicted pose data generated by an object prediction node. Additionally, the simulation engine also receives and utilizes accuracy information indicating the accuracy of the pose data and the predicted pose data. Based on this accuracy data, the simulation engine makes processing decisions for the AR / MR data.

[0011] Accordingly, in a first aspect, the present disclosure provides a method, implemented in an object detector node of a communications network, of processing Augmented Reality (AR)ZMixed Reality (MR) data. In this aspect, the method comprises detecting a pose of an object based on sensor data provided by one or more sensors and generating pose output data for the pose. The pose output data comprises pose data indicating a detected position and a detected orientation of the object in space, as well as pose accuracy data. The pose accuracy data indicates the accuracy of the pose data relative to one of actual pose data indicating an actual position and an actual orientation of the object in space and previously detected pose data. Once generated, the method comprises sending the pose output data to one or more nodes in the communications network.

[0012] In a second aspect, the present disclosure provides an object detector node in a communications network for processing Augmented Reality (AR) / Mixed Reality (MR) data. In this aspect, the object detector node is configured to detect a pose of an object based on sensor data provided by one or more sensors and generate pose output data for the pose. The pose output data comprises pose data indicating a detected position and a detected orientation of the object in space and pose accuracy data indicating an accuracy of the pose data relative to one of actual pose data that indicates an actual position and an actual orientation of the object in space and previously detected pose data. The object detector node is also configured in this aspect to send the pose output data to one or more nodes in the communications network.

[0013] In a third aspect, the present disclosure provides an object detector node in a communications network for processing Augmented Reality (AR) / Mixed Reality (MR) data. The object detector node comprises processing circuitry and memory configured to store instructions executable by the processing circuitry. When the instructions are executed by the processing circuitry, the processing circuitry is configured to detect a pose of an object based on sensor data provided by one or more sensors and generate pose output data for the pose. The pose output data comprises pose data indicating a detected position and a detected orientation of the object in space and pose accuracy data indicating an accuracy of the pose data relative to one of actual pose data that indicates an actual position and an actual orientation of the object in space and previously detected pose data. The object detector node is also configured in this aspect to send the pose output data to one or more nodes in the communications network.

[0014] In a fourth aspect, the present disclosure provides a computer program comprising instructions that, when executed on processing circuitry of an object detector node configured to process Augmented Reality (AR) / Mixed Reality (MR) data, causes the object detector node to perform the method according to the first aspect.

[0015] In a fifth aspect, the present disclosure provides a non-transitory computer-readable storage medium comprising a computer program stored thereon. The computer program comprises executable instructions that, when executed by processing circuitry in an object detector node configured to process Augmented Reality (AR) / Mixed Reality (MR) data, causes the object detector node to perform the method according to the first aspect.

[0016] In a sixth aspect, the present disclosure provides a method, implemented in an object predictor node in a communications network, of processing Augmented Reality (AR) / Mixed Reality (MR) data. In this aspect, the method comprises the object predictor node receiving, from an object detector node, pose output data for an object detected by the object detector node. The pose output data comprises pose data indicating a detected position and a detected orientation of the object in space and pose accuracy data indicating an accuracy of the pose data relative to one of actual pose data that indicates an actual position and an actual orientation of the object in space and previously detected pose data. The method also calls for the object predictor node generating, based on the pose output data, predicted pose output data comprising predicted pose data indicating a predicted position and a predicted orientation of the object in space and prediction accuracy data indicating an accuracy of the predicted pose data. The prediction accuracy data indicates an accuracy of the predicted pose data relative to one of the actual pose data that indicates the actual position and the actual orientation of the object in space and previously predicted pose data. So generated, the method further comprises the object predictor node sending the predicted pose output data to one or more nodes in the communications network.

[0017] A seventh aspect of the present disclosure provides an object predictor node in a communications network for processing Augmented Reality (AR) / Mixed Reality (MR) data. The object predictor node is configured to receive, from an object detector node, pose output data for an object detected by the object detector node. The pose output data comprises pose data indicating a detected position and a detected orientation of the object in space and pose accuracy data indicating an accuracy of the pose data relative to one of actual pose data that indicates an actual position and an actual orientation of the object in space and previously detected pose data. The object predictor node is further configured to generate, based on the pose output data, predicted pose output data comprising predicted pose data indicating a predicted position and a predicted orientation of the object in space and prediction accuracy data indicating an accuracy of the predicted pose data. The accuracy of the predicted pose data is relative to one of the actual pose data that indicates the actual position and the actual orientation of the object in space and previously predicted pose data. The object predictor node is also configured to send the predicted pose output data to one or more nodes in the communications network.

[0018] An eighth aspect of the present disclosure provides an object predictor node in a communications network for processing Augmented Reality (AR) / Mixed Reality (MR) data. In this aspect, the object predictor node comprises processing circuitry and memory configured to store instructions executable by the processing circuitry. When executed by the processing circuitry, the instructions configure the processing circuitry to receive, from an object detector node, pose output data for an object detected by the object detector node. The pose output data comprises pose data indicating a detected position and a detected orientation of the object in space and pose accuracy data indicating an accuracy of the pose data relative to one of actual pose data that indicates an actual position and an actual orientation of the object in space and previously detected pose data. The processing circuitry is further configured to generate, based on the pose output data, predicted pose output data comprising predicted pose data indicating a predicted position and a predicted orientation of the object in space and prediction accuracy data indicating an accuracy of the predicted pose data. The accuracy of the predicted pose data is relative to one of the actual pose data that indicates the actual position and the actual orientation of the object in space and previously predicted pose data. The processing circuitry is also configured to send the predicted pose output data to one or more nodes in the communications network.

[0019] A ninth aspect of the present disclosure provides a computer program comprising instructions that, when executed on processing circuitry of an object predictor node configured to process Augmented Reality (AR) / Mixed Reality (MR) data, causes the object predictor node to perform the method according to the sixth aspect.

[0020] In a tenth aspect, the present disclosure provides a non-transitory computer-readable storage medium comprising a computer program stored thereon, the computer program comprising executable instructions that, when executed by processing circuitry in an object predictor node configured to process Augmented Reality (AR) / Mixed Reality (MR) data, causes the object predictor node to perform the method of the sixth aspect. An eleventh aspect of the present disclosure provides a method, performed by a network simulator node implementing a simulation engine in a communications network, of processing Augmented Reality (AR) / Mixed Reality (MR) data. In this aspect, the method comprises the network simulator node receiving pose data output from an object detector node and / or prediction data output from an object predictor node. The pose data output comprises pose data indicating a detected position and a detected orientation of the object in space and pose accuracy data indicating an accuracy of the pose data relative to one of actual pose data that indicates an actual position and an actual orientation of the object in space and previously detected pose data. The predicted pose output data comprises predicted pose data indicating a predicted position and a predicted orientation of the object in space and prediction accuracy data indicating an accuracy of the predicted pose data relative to one of the actual pose data that indicates the actual position and the actual orientation of the object in space and previously predicted pose data. The method of this aspect further comprises the network simulator node updating a simulation component of a world state of the simulation engine based on the pose accuracy data and the prediction accuracy data and sending AR / MR data to a user for display based on the updated one or more simulation components of the world state.

[0021] A twelfth aspect of the present disclosure provides a network simulator node in a communications network configured to implement a simulation engine for processing Augmented Reality (AR)ZMixed Reality (MR) data. The network simulator node is configured in this aspect to receive pose data output from an object detector node and prediction data output from an object predictor node. The pose data output comprises pose data indicating a detected position and a detected orientation of the object in space and pose accuracy data indicating an accuracy of the pose data relative to one of actual pose data that indicates an actual position and an actual orientation of the object in space and previously detected pose data. The predicted pose output data comprises predicted pose data indicating a predicted position and a predicted orientation of the object in space and prediction accuracy data indicating an accuracy of the predicted pose data relative to one of the actual pose data that indicates the actual position and the actual orientation of the object in space and previously predicted pose data. The network simulator node of this aspect is further configured to update a simulation component of a world state of the simulation engine based on the pose accuracy data and the prediction accuracy data and send AR / MR data to a user device for display based on the updated one or more simulation components of the world state.

[0022] In a thirteenth aspect, the present disclosure provides a network simulator node in a communications network for processing Augmented Reality (AR) / Mixed Reality (MR) data. The network simulator node comprises processing circuitry and memory configured to store instructions executable by the processing circuitry. Executing the instructions configures the processing circuitry to receive pose data output from an object detector node and prediction data output from an object predictor node. The pose data output comprises pose data indicating a detected position and a detected orientation of the object in space and pose accuracy data indicating an accuracy of the pose data relative to one of actual pose data that indicates an actual position and an actual orientation of the object in space and previously detected pose data. The predicted pose output data comprises predicted pose data indicating a predicted position and a predicted orientation of the object in space and prediction accuracy data indicating an accuracy of the predicted pose data relative to one of the actual pose data that indicates the actual position and the actual orientation of the object in space and previously predicted pose data. Executing the instructions further configures the processing circuitry to update a simulation component of a world state of the simulation engine based on the pose accuracy data and the prediction accuracy data and send AR / MR data to a user device for display based on the updated one or more simulation components of the world state.

[0023] In a fourteenth aspect, the present disclosure provides a computer program comprising instructions that, when executed on processing circuitry of a network simulator node configured to process Augmented Reality (AR) / Mixed Reality (MR) data, cause the network simulator node to perform the method according to the eleventh aspect.

[0024] A fifteenth aspect of the present disclosure provides a non-transitory computer-readable storage medium comprising a computer program stored thereon. The computer program comprises executable instructions that, when executed by processing circuitry in a network simulator node configured to process Augmented Reality (AR) / Mixed Reality (MR) data, causes the network simulator node to perform the method of the eleventh aspect.

[0025] A sixteenth aspect of the present disclosure provides a system configured to process Augmented Reality (AR) / Mixed Reality (MR) data. In this aspect, the system comprises an object detector node, an object predictor node, and a network simulator node. The object detector node is configured to generate pose output data for a pose of an object in space, wherein the pose output data comprises pose data indicating a detected position and a detected orientation of the object in space and pose accuracy data indicating an accuracy of the pose data. The accuracy of the pose data is relative to one of actual pose data that indicates an actual position and an actual orientation of the object in space and previously detected pose data. The object predictor node is configured to generate, based on the pose output data, predicted pose output data comprising predicted pose data indicating a predicted position and a predicted orientation of the object in space and prediction accuracy data indicating an accuracy of the predicted pose data. The accuracy of the predicted pose data is relative to one of the actual pose data that indicates the actual position and the actual orientation of the object in space and previously predicted pose data. The network simulator node is configured to update a simulation component of a world state of a simulation engine based on the pose accuracy data and the prediction accuracy data and send AR / MR data to a user device for display based on the updated one or more simulation components of the world state.

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart illustrating some functional components of a method for processing AR / MR data.

[0028] Figures 2-3 are flow charts illustrating methods for processing AR / MR data according to aspects of the present disclosure.

[0029] Figure 4 is a flow diagram illustrating a method, implemented at an object detector node, for detecting the pose of a real-world object according to aspects of the present disclosure.

[0030] Figure 5 is a flow diagram illustrating a method, implemented at an object predictor node, for predicting a future pose of the real-world object according to aspects of the present disclosure.

[0031] Figure 6 is a flow diagram illustrating a method, performed at a network node implementing a simulation engine, for simulating the real-world and / or virtual world according to aspects of the present disclosure.

[0032] Figure 7 is a functional block diagram illustrating a network node configured to process AR / MR data according to aspects of the present disclosure.

[0033] Figure 8 is a functional block diagram illustrating an example of a communication system in accordance with some embodiments.

[0034] Figure 9 is a functional block diagram illustrating an example User Equipment (UE) in accordance with some embodiments.

[0035] Figure 10 is a functional block diagram illustrating an example network node in accordance with some embodiments.

[0036] Figure 11 is a functional block diagram of a host, which may be an embodiment of the host illustrated in Figure 8, in accordance with some embodiments described herein.

[0037] Figure 12 is a functional block diagram illustrating a virtualization environment in which functions implemented by some embodiments of the present disclosure may be virtualized.

[0038] Figure 13 is a functional block diagram illustrating a communication diagram of a host communicating via a network node with a UE over a partially wireless connection in accordance with some embodiments.

[0039] DETAILED DESCRIPTION

[0040] Figure 1 illustrates a method 10 showing some of the key functional component processes implemented by example AR application. As seen in method 10, one or more sensors, such as cameras and Inertial Measurement Units (IMUs) in a User Equipment (UE) send data (e.g., 2D / 3D images, camera poses, a timestamp indicating when the sensor captured the data, such as an image) to an object detection function (box 12). The UE may be, for example, a user’s mobile device or a Head Mounted Display (HMD). The object detection function then detects the pose of a moving real object from the sensor data (box 14). As described herein, the “pose” of an object comprises data representing a position and an orientation of a detected object.

[0041] In some embodiments, the object detection function comprises both an object detection function and an object tracking function. The object detection function will return the identifier of an object (i.e., the Object ID) detected in a frame, as well as the location and orientation of the sensor (e.g., a camera) relative to the detected object. The tracking function tracks the movements of the detected object in space and / or across different camera angles. However, for the sake of simplicity and ease of discussion, this disclosure refers to both the detection and the tracking functions as object detection implemented by an object detector node.

[0042] Once the object detector node has identified an object, it sends the detected pose of the object to an object predictor node. The object predictor node then predicts a future pose of the detected object for some future timestamp and sends data representing the predicted pose to a simulation engine, such as a game engine, for example (box 16). The predicted pose data comprises a predicted future position and a predicted future orientation for the detected object, as well as the future timestamp indicating when the object is predicted to be in the predicted position and predicted orientation. Upon receipt, the simulation engine determines, according to the laws of physics, the pose of a virtual object (e.g., a ball) based on the pose(s) of the real object (e.g., a real ball moving through space) (box 18). More particularly, the simulation engine performs application logic in addition to the simulation logic, as well as client specific functions. For example, the simulation engine can be configured to render images for output to a client display. Finally, the simulation engine sends client-specific updates (e.g., AR / MR data, a rendered image, etc.) to a display associated with the user (box 20).

[0043] As stated above, the object detection function detects the current pose of a moving real object from the sensor data, while the object prediction function predicts the future pose of the moving object. There are various conventional ways to predict the pose(s) of a detected object. One method, for example, is provided in U.S. Pat. No. 10,943,379 in which a motion predictor predicts the future position and orientation of a real object. Another method for predicting the pose of a detected object is described in U.S. Pat. No. 10,962,780. This method discloses remote rendering for virtual images, in which a pose prediction machine is configured to predict a future pose of the virtual reality device. A third method is described in U.S. Pat. No. 11 ,043,018, which provides a video pipeline where head motion prediction is part of the system.

[0044] Although useful, conventional methods for detecting objects and predicting their movement can be problematic. For example, object detection errors can negatively affect the accuracy and quality of the pose data being generated for an object. Such object detection errors may be caused by, or related to, image quality problems. Examples of object detection errors include, but are not limited to, camera issues such as focus or light condition adaptation, target issues such as image blur due to object movement, external disruptions such as when other objects “hide” or obscure the target object, and video artifacts caused by network packet losses. Similarly, prediction errors can also negatively affect the accuracy and quality of predicted pose data. Some exemplary causes of prediction errors include, for example, noisy data, predictions that are too far ahead in the future, and the difficulties in predicting the random movement of an object. Regardless of the reasons, however, each of these can be the cause of low-quality object detection pose data and low-quality prediction pose data.

[0045] Additionally, for real-time interactive AR applications (e.g., applications in which real and virtual objects interact), the environment information is usually lacking. Particularly, due at least partially to the time required for detection, the results from detecting an object are delayed. That is, by time the information representing the current position and orientation of an object is processed and provided to a simulation engine (or an object predictor node), the pose data for the detected object is already “old.” As for object prediction, information from the object predictor node in a generic case is unreliable. This is because the information provided about the predicted pose of the object is based on assumptions and speculations about the future, which may or may not be correct.

[0046] Unfortunately, the decisions based on low quality or inappropriate data can cause various data inaccuracies and misalignments in the virtual world state maintained in the simulation engine. Such inaccuracies and misalignments can seriously affect the UE and the usability of some or all of the AR application / world. By way of example only, consider an AR application for playing a soccer game. In such situations, predicting the future pose of a real- world object (e.g., a soccer ball) based on inaccurate data may cause the AR application to reflect an incorrect goal count. Similarly, an incorrect “hit” count based on a predicted “hit” for a user in an AR fighting game can render an incorrect game result.

[0047] Accordingly, embodiments of the present disclosure address such inaccuracies and data misalignments in AR / MR scenarios where an accurate, up-to-date pose of a real object is needed for a simulation engine. More particularly, embodiments of the present disclosure configure a network node to execute a simulation engine. As is conventional, the simulation engine of the present disclosure receives and utilizes both the delayed but usually accurate pose data generated by a network node implementing an object detection component, and the more current but less accurate predicted pose data generated by a network node implementing an object prediction component. However, in addition to these conventional functions, a simulation engine configured according to the present embodiments also receives and utilizes accuracy information representing the accuracy of the pose data generated by the network node implementing the object detection function, and the predicted pose data generated by the network node implementing the object prediction function. The simulation engine is configured to access multiple versions of the same data. In at least one embodiment, the simulation engine is made aware that it is using data that is either late (i.e., delayed) but accurate, less accurate, or predicted. In doing so, the present embodiments enable AR / MR applications to intentionally delay or postpone making applicationlevel decisions or responding to events depending on the “real” environment (i.e., the pose of the detected real object) until it obtains pose data that has a high / good enough accuracy. This accuracy data may, for example, comprise metadata that is generated by the object detection component and the object prediction component, and sent to the simulation engine along with the pose data and the predicted pose data.

[0048] Thus, the present disclosure configures the simulation engine to know that it is processing accurate but not- so- current pose data for a detected object, and predicted pose data that may be more current but not fully accurate and reliable. Then, based on this information, a simulation engine configured according to the present embodiments can consider this information when making a decision or when announcing decisions. For example, in one embodiment, the simulation engine can delay certain app level decisions until the detected or predicted object pose accuracy becomes high enough. To accomplish this, the present disclosure modifies the object detection and object prediction components to send, respectively, the pose data and the predicted pose data to the simulation engine along with detection accuracy metadata and prediction accuracy metadata indicating the accuracy of the pose data and the prediction pose data.

[0049] Configuring the network node(s) implementing the object detection component, the object prediction component, and the simulation engine provide advantages and benefits that conventional AR / MR processing systems cannot or do not provide. For example, conventional systems implicitly assume all generated pose data is “100%” accurate. Embodiments of the present disclosure, however, do not. Rather, the present embodiments configure the network nodes to determine how accurate the pose data and the predicted pose data are, and then explicitly includes the “accuracy data” in its messages to the simulation engine.

[0050] In doing so, the present disclosure configures a simulation engine to avoid making incorrect decisions based on low quality predicted pose data. Beneficially, avoiding such decisions prevents the network node from having to implement revoke / rollback procedures. Additionally, the results of a wrong decision could be visible to the simulation engine for an extended period of time, thereby increasing the chances that the simulation engine will continue to make bad decisions. Further, the results of a wrong decision may be too complex to revert. For example, consider a game in which a virtual object explodes due to a bad decision made by the simulation engine. In these cases, it would be too complex to revert or rollback the effects of the bad decision (i.e., the explosion) after-the-fact. As another example, consider an AR / MR soccer game that updates the game result counter (i.e., the score) only when the ball is definitely behind the goal line. In these cases, a wrong decision (e.g., determining that a ball is behind the goal line when it is not) could start a series of wrong events. For example, game play would resume with the ball in an incorrect location on a virtual field even though a goal was not scored. These types of events are much more difficult to rollback and require much more time to rollback. Similarly, when based on a wrong event or decision, events that result from user interactions can be quite problematic to undo. However, with a simulation engine configured according to the present embodiments, however, such difficulties and situations can be easily avoided.

[0051] Additionally, according to the present embodiments, AR / MR application developers are able to decide how to use the accuracy information. By way of example only, AR / MR application developers may decide to directly use predicted pose information (i.e., inaccurate but not delayed) to display the object, while using the more accurate but delayed detected pose information for more important decisions.

[0052] Further, the embodiments of the present disclosure are backward compatible. That is, using the accuracy information generated by the object detection and object prediction components is optional. As such, legacy AR / MR applications are able to function with a simulation engine configured according to the present embodiments without having to utilize the added accuracy information.

[0053] Another advantage is that the present disclosure configures the simulation engine to handle the detected pose data and the predicted pose data in the same abstract manner. That is, the object detection and prediction components are each configured to send the accuracy information in a message to the simulation engine. In one embodiment, such an output message follows the following format:

[0054] [T, ObjectID, PData, AccuracyData] where:

[0055] • T is a timestamp associated with the detected pose or the predicted pose of an object. For instance, consider a situation where the sensor is a camera. In these cases, a timestamp TDwould indicate when the camera captured the image of the object.

[0056] Alternatively, a timestamp TPwould indicate when the prediction component generated the predicted pose data;

[0057] • ObjectID is the ID of the detected object;

[0058] • PData is either the pose data generated by the object detection component or the predicted pose data generated by the object prediction component; and

[0059] • AccuracyData is the information indicating the accuracy of PData.

[0060] Additionally, a source indicator may be sent along with the output message indicating the source of the accuracy information. In one embodiment, for example, the source indicator is part of the AccuracyData or PData fields in the output message. In another embodiment, the source indicator is sent in the output message as its own field. In one embodiment, the output message may identify T as being either TDor TP. In such cases, the simulation engine would be informed based on the timestamp whether the object detection component or the object prediction component is the source of the PData and the AccuracyData. Regardless of the particular implementation, however, the source indicator of the present disclosure may comprise multiple bits or a single bit and informs the simulation engine as to whether it is working with detected pose data or predicted pose data.

[0061] In another embodiment, the simulation engine is configured to assume by default that the object detection component detected an object with 100% accuracy. However, in some cases, such as when an object is partially hidden or obscured by another object, the detection accuracy may be significantly lower (e.g., 60%). Similarly, the object prediction component may only be partially accurate (e.g., 80%) by default. Nevertheless, in at least some situations, the object prediction component may be much more accurate (e.g., 95%).

[0062] In some embodiments, the particular algorithm(s) and / or the algorithm parameters used by the object detection component and / or the object prediction component to generate their respective pose data is sent to the simulation engine as metadata in the output message. As such, the simulation engine can analyze and act on the metadata at a later time.

[0063] In yet another embodiment of the present disclosure, the AR / MR application may know the context of the environment and evaluate the detection / prediction quality information (i.e., the AccuracyData) according to that knowledge. For example, consider the virtual soccer game discussed above. If there are no players around the user, then the movement and path of the soccer ball is predictable. However, the movement and path of the soccer ball is not so predictable in cases where there are many users and possible interactions. Therefore, in some embodiments, the simulation engine may determine that 80% prediction quality is sufficient (or barely usable depending on the context).

[0064] Figure 2 is a flow chart illustrating methods 30, 40 for processing AR / MR data at an object detector node (i.e., method 30) and an object predictor node (i.e., method 40) according to aspects of the present disclosure. In this embodiment, methods 30 and 40 extend the example pipeline seen in Figure 1 by generating accuracy data in both the object detection process and the object prediction process.

[0065] More particularly, method 30 begins with the object detector node detecting an object from the data provided by one or more sensors, as previously described (box 32). As seen in method 30, the detection process results in the generation of a detector output message. In this embodiment, the detector output message includes, but is not limited to, a timestamp provided by the sensor that sent the sensor data, the Object ID of the detected object, and the pose data, which as previously described, comprises data representing the current position and the current orientation of the detected object.

[0066] Additionally, the object detector node also generates detected accuracy information (i.e., quality information) indicating the accuracy of the pose data (box 34). According to the present disclosure, the detected accuracy information is included in the detector output message that is sent to the object predictor node. In at least one embodiment, the detector output message and / or the detected accuracy information is also sent to a simulation engine which, as seen later in more detail, implements a method 50 for processing the data generated by the object detector node and / or the object predictor node in methods 30, 40, respectively.

[0067] The detector output message can be generated according to any format or structure needed or desired. However, in one embodiment, the detector output message comprises the following elements:

[0068] • A timestamp indicating when the sensor captured the sensor data associated with the detected object;

[0069] • An Object ID uniquely identifying the detected object;

[0070] • Pose data representing the current position and orientation of the detected object in space; and

[0071] • Detected Accuracy Metadata (i.e., the accuracy information).

[0072] Upon receipt of the detector output message at the object predictor node (method 40), the object predictor node generates predicted pose data for the detected object (box 42). The predicted pose data comprises information representing, for example, a predicted position and a predicted orientation for the detected object at some future point in time. This future point in time, represented in Figure 2 as the predicted timestamp, is relative to the timestamp derived by the object detector node. Then, the object predictor node generates accuracy information indicating the quality of the predicted pose data (box 44), and generates a predicted pose output message that includes, but is not limited to, the following data.

[0073] • A predicted timestamp indicating when the detected object is predicted to be in the predicted position and the predicted orientation;

[0074] • The Object ID uniquely identifying the detected object;

[0075] • Predicted pose data representing the predicted position and the predicted orientation of the object in space at the predicted timestamp; and

[0076] • Prediction Accuracy Metadata (i.e., the predicted accuracy information).

[0077] The accuracy metadata (i.e., the accuracy data generated by one or both of the object detector node and the object predictor node) may comprise any data needed or desired. However, in one embodiment, such accuracy metadata may comprise some or all of the following data: • An indication that identifies the source of the pose output / predicted pose output. In at least one aspect, the indication may be encoded as a single bit of information that indicates whether the data comes from an object detector node or an object predictor node. For example, a ‘0’ bit may be used to indicate that the accuracy metadata is received from the object detector node, while a ‘1 ’ bit may be used to indicate that the accuracy metadata is received from the object predictor node.

[0078] • An indicator representing the accuracy of the pose data / predicted pose data. According to the present embodiments, the accuracy can be expressed in various ways. By way of example, the accuracy metadata in one embodiment includes a value that indicates the accuracy of the data as a percentage relative to a most accurate pose. This value may be an abstract value or it can have a predetermined physical meaning.

[0079] • A Circular Error Probable (CEP) value representing the radius of a sphere with the pose data / predicted pose data at the center of the sphere. If the actual position of the detected object lies within the sphere, then the position of the detected object is within the 50th percentile of the positions of other objects proximate the actual position of the detected object. The CEP value, therefore, is a measure of the accuracy of the position of the detected object relative to the actual position of the detected object. The reverse is also true, in which the actual position of the detected object is at the center of the sphere and the detected or predicted position of the detected object is within the sphere.

[0080] • A confidence interval of the pose (e.g., an interval indicating a confidence level that the real / predicted position and / or orientation of the detected object is within a predetermined distance to the detected / predicted position and / or orientation with a given percentage. By way of example only, a confidence interval of “0.1 meter with 98%” might mean that the node generating the confidence interval (i.e., the object detector node or the object predictor node) is 98% certain that the detected or predicted position of the detected object is within 0.1 meters of the actual position of the detected object; and

[0081] • Additional information on how the object detector node and / or the object predictor node determined the accuracy metadata. By way of example, such information can be encoded as being the particular algorithm used by the object detector node or the object predictor node for detection / prediction (e.g., a 2nd degree curve fitting algorithm). In other embodiments, this information is encoded with one or more parameters used for the algorithm. Such parameters include, but are not limited to, 10 points, 1.4 compliance ratio, 100ms prediction time. etc..

[0082] Those of ordinary skill in the art should understand that some or all of the foregoing examples can be combined and indicated in a pose output message or a predicted pose output message. Figure 3 is a flow chart illustrating a method 50 for processing AR / MR data at a simulation engine according to aspects of the present disclosure. In this embodiment, method 50 uses the accuracy metadata generated in one or both of the object detector node and the object predictor node when performing the decision logic.

[0083] As an example, the simulation engine of Figure 3 implements a soccer game. In this game, the pose (i.e., the position and orientation of the ball) of a physical, real-world soccer ball is represented in the “world state” of the simulation engine as a virtual ball. During game play, the simulation engine of the present disclosure determines whether or not the physical soccer ball crosses a virtual goal line. If so, the simulation engine updates the game result (i.e., the score) to a virtual scoreboard object. Additionally, the simulation engine renders the virtual ball on top of the physical, real-world soccer ball, for example, to display the speed of the ball.

[0084] As seen in method 50, the simulation engine receives the pose output data (e.g., in a message) and the predicted pose data output (e.g., in the same or another message) from the object detector node and the object predictor node, respectively. As previously described, these messages were generated in methods 30 and 40 (see Figure 2) and carry the accuracy metadata generated by the object detector node and the object predictor node. Additionally, the simulation engine implements decision logic (box 52) that uses the accuracy metadata provided by one or both of the object detector node and the object predictor node to select which pose update to send to which virtual object 54, 56, 58 of the world state.

[0085] For example, given the accuracy data and the pose data generated by the object detector node, which is considered to be the most accurate data, the decision logic might send the corresponding pose update to the virtual scoreboard component (box 54). However, in some cases, the detected real-world object may be at least partially obscured by one or more other objects. In such scenarios, the accuracy metadata generated by the object detector node may indicate only 60% accuracy. Therefore, the decision logic 52 may decide to update the virtual object component 56 in the world state using the predicted pose data and the predicted accuracy.

[0086] For instance, consider a situation where the accuracy metadata generated by the object detector node indicates that the pose data is 60% accurate. Additionally, consider where the accuracy metadata generated by the object predictor node indicates that the predicted pose data is about 85% accurate. In these cases, the simulation engine might be configured to update the virtual object component according to the predicted pose data (box 56).

[0087] In some cases, there may be other virtual object components (box 58). In these cases, the decision logic (box 52) may be configured for a particular preference between delay and accuracy. For example, in the embodiment of Figure 3, the decision logic (box 52) may update a virtual object component (box 58) with the pose data (or the predicted pose data) in cases where the accuracy of the pose data (or the predicted pose data) meets or exceeds a predetermined threshold value of X%.

[0088] Thus, the simulation engine of this particular embodiment comprises three components - i.e., the virtual scoreboard component (box 54), the first virtual object component (box 56), and the second virtual object component (box 58), which is different from the first virtual object component (box 56). In this embodiment, each component is updated based on the accuracy of the pose data (or predicted pose data). Specifically, the virtual scoreboard component (box 54) requires the most accurate pose information, even in cases where the information is delayed compared to reality. An example would be the score of the soccer game. As stated above, the simulation engine should update the score of the game only in situations where the ball actually crossed the goal line. The virtual scoreboard component (box 54) therefore consumes the detected real ball pose updates (i.e., the pose data provided by the object detector node).

[0089] The first virtual object component (box 56) may need the latest, most current pose data for the ball even if it is not entirely accurate. As stated above, such pose data is provided by the object predictor node as the predicted pose data. For example, consider a virtual ball object representing the real ball object. The virtual ball object should move in real time and synchronously with the real ball object. This (predicted) virtual ball can be rendered and displayed to users, although the predicted pose may sometimes be spatially inaccurate relative to the real pose of the ball. However, the preference in this embodiment is to be time- synchronous with reality. Thus, the first virtual object component (box 56) consumes the predicted pose data updates.

[0090] A variety of other object components, such as the second virtual object component (box 58), process pose data according to other preferences. As previously described, for example, the second virtual object (box 58) may be configured for a particular preference between delay and accuracy. Thus, in this embodiment, the second virtual object component (box 58) may be configured to consume the pose updates generated by the object detector node or the object predictor node in cases where the accuracy of that pose (i.e., the pose data generated by the object detector node or the predicted pose data generated by the object predictor node) meets or exceeds a predetermine threshold of X%.

[0091] Regardless of which component consumes the updates, however, the simulation engine can, in at least some embodiments, render the detected object (box 60) before sending the rendered object to a display device (e.g., a user’s HMD) for display to the user (box 62).

[0092] It should be noted here that the object detector node, the object predictor node, and the simulation engine may all be implemented on the same physical node (e.g., a network node in the cloud). Alternatively, however, the object detector node, the object predictor node, and the simulation engine may be distributed across multiple network nodes in the cloud. Regardless of the number of physical nodes that are configured to implement the object detector node, the object predictor node, and the simulation engine, however, the simulation engine beneficially utilizes accuracy information generated by, and sent by, the object detector node and the object predictor node to determine which of a plurality of components in the world state to update with the received pose data or the received predicted pose data.

[0093] Figure 4 is a flow chart illustrating a method 70 of processing AR / MR data at an object detector node according to one embodiment of the present disclosure. As seen in Figure 4, the object detector node detects a pose of an object based on sensor data provided by one or more sensors (box 72). Then, based on the sensor data, the object detector node generates pose output data for the pose (box 74). In this embodiment, the pose output data is generated to include pose data indicating a detected position and a detected orientation of the object in space and pose accuracy data indicating an accuracy of the pose data. In this embodiment, the pose accuracy data indicates the accuracy of the pose data relative to one of actual pose data that indicates an actual position and an actual orientation of the object in space, and previously detected pose data. In some embodiments, the object detector node encodes information that specifies an algorithm that was used by the object detector node to detect the object and / or one or more parameters used by the algorithm to detect the object (box 76). Regardless, though, the object detector node sends the pose output data to one or more nodes in the communications network (box 78).

[0094] In one embodiment, the pose accuracy data comprises a pose accuracy indicator indicating a type of network node as being the source of the pose data. In such embodiments, the type of node is one of an object detector and an object predictor.

[0095] In one embodiment, the pose accuracy data comprises a Circular Error Probable (CEP) value representing whether the actual position and / or the actual orientation of the object is within a predetermined percentile of the detected position and / or the detected orientation of the object.

[0096] In one embodiment, the pose accuracy data comprises a confidence value indicating a percentage that the actual position of the object and the detected position of the object are within a predetermined distance of each other.

[0097] In one embodiment, the pose accuracy data comprises a percentage value indicating the accuracy of the pose data relative to previously detected pose data.

[0098] In one embodiment, the one or more nodes in the communications network comprise a pose predictor node configured to predict a future pose of the object.

[0099] In one embodiment, the one or more nodes in the communications network comprise a simulator node configured to implement a simulation engine.

[0100] Figure 5 is a flow chart illustrating a method 80 of processing AR / MR data at an object predictor node according to one embodiment of the present disclosure. As seen in Figure 5, the object predictor node first receives, from an object detector node, pose output data for an object detected by the object detector node (box 82). In this embodiment, the pose output data comprises pose data indicating a detected position and a detected orientation of the object in space and pose accuracy data indicating an accuracy of the pose data. The pose accuracy data is relative to one of actual pose data that indicates an actual position and an actual orientation of the object in space and previously detected pose data. Then, based on the received pose output data, the object predictor node generates predicted pose output data (box 84). As seen in the embodiment of Figure 5, the predicted pose output data comprises predicted pose data indicating a predicted position and a predicted orientation of the object in space, and prediction accuracy data indicating an accuracy of the predicted pose data relative to one of the actual pose data that indicates the actual position and the actual orientation of the object in space and previously predicted pose data. In some embodiments, the object predictor node encodes information that specifies an algorithm that was used by the object predictor node to predict the future pose of the object and / or one or more parameters used by the algorithm to perform the prediction (box 86). The object predictor node then sends the predicted pose output data to one or more nodes in the communications network (box 88).

[0101] In one embodiment, the prediction accuracy data comprises a prediction accuracy indicator indicating a type of network node as being the source of the predicted pose data. The type of node may be, according to the present disclosure, one of an object detector and an object predictor.

[0102] In one embodiment, the prediction accuracy data comprises a Circular Error Probable (CEP) value representing whether the actual position and / or the actual orientation of the object is within a predetermined percentile of the predicted position and / or the predicted orientation of the object.

[0103] In one embodiment, the prediction accuracy data comprises a confidence value indicating a percentage that the actual position of the object and the predicted position of the object are within a predetermined distance of each other.

[0104] In one embodiment, the prediction accuracy data comprises a percentage value indicating the accuracy of the predicted pose data relative to previously predicted pose data.

[0105] In one embodiment, the one or more nodes in the communications network comprise a simulator node configured to implement a simulation engine.

[0106] Figure 6 is a flow chart illustrating a method 90 of processing AR / MR data at a simulator node implementing a simulation engine according to one embodiment of the present disclosure. As seen in Figure 6, the simulator node receives pose data output from an object detector node and / or prediction data output from an object predictor node (box 92). The pose data output comprises pose data indicating a detected position and a detected orientation of the object in space and pose accuracy data indicating an accuracy of the pose data. The accuracy of the pose data in this embodiment relative to one of actual pose data that indicates an actual position and an actual orientation of the object in space and previously detected pose data.

[0107] As for the predicted pose output data, it comprises predicted pose data indicating a predicted position and a predicted orientation of the object in space and prediction accuracy data indicating an accuracy of the predicted pose data relative to one of the actual pose data indicating the actual position and the actual orientation of the object in space and previously predicted pose data.

[0108] Then, based on the pose accuracy data and the prediction accuracy data, the simulator node updates a simulation component of a world state of the simulation engine (box 94). So updated, the simulator node can either send AR / MR data to a user for display based on the updated one or more simulation components of the world state (box 96) or render an image for display to the user based on the updated one or more simulation components of the world state (box 98). In the latter case, the simulator node also outputs the rendered image to a display device (e.g., an HMD) for the user (box 100).

[0109] In one embodiment, the simulator node also determines which of the one or more simulation components of the world state to update based on the pose accuracy data and the prediction accuracy data.

[0110] In one embodiment, updating a simulation component of a world state comprises updating an object pose of the object according to the pose data.

[0111] In one embodiment, updating a simulation component of a world state comprises updating an object pose of the object according to the predicted pose data responsive to determining that the pose accuracy data does not meet or exceed a first predetermined threshold value, and the predicted accuracy data meets or exceeds a second predetermined threshold value.

[0112] In one embodiment, updating a simulation component of a world state comprises updating an object pose of the object according to the pose data responsive to determining that the pose accuracy data meets or exceeds a predetermine threshold value.

[0113] In one embodiment, updating a simulation component of a world state comprises updating an object pose of the object according to the predicted pose data responsive to determining that the predicted pose accuracy data meets or exceeds a predetermine threshold value.

[0114] In one embodiment, one or more decisions made by the simulation engine are delayed or postponed based on an accuracy of the pose data.

[0115] In one embodiment, the previously detected pose data indicates a previously detected position and a previously detected orientation of the object in space and is stored as being the most accurate detected pose data. An apparatus can perform any of the methods herein described by implementing any functional means, modules, units, or circuitry. In one embodiment, for example, the apparatuses comprise respective circuits or circuitry configured to perform the steps shown in the method figures. The circuits or circuitry in this regard may comprise circuits dedicated to performing certain functional processing and / or one or more microprocessors in conjunction with memory. For instance, the circuitry may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include Digital Signal Processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory may include program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein, in several embodiments. In embodiments that employ memory, the memory stores program code that, when executed by the one or more processors, carries out the techniques described herein.

[0116] Figure 7 is a functional block diagram illustrating some of the main functional components of a network node 200 configured to process AR / MR data according to embodiments of the present disclosure. As seen in Figure 7, network node 200 comprises a physical computing node having physical circuitry and can be configured to implement one or more of the object detector node, the object predictor node, and the simulator node functions, as previously described. In some cases, there are multiple network nodes 200 configured to implement one or more of the object detector node, the object predictor node, and the simulator node. Thus, according to the present disclosure, there may be multiple nodes configured to execute these functions that are distributed throughout the communication network (e.g., the cloud).

[0117] As seen in Figure 7, network node 200 comprises communication circuitry 202, processing circuitry 204, and memory 206. The communication circuitry 202 comprises the hardware required for communicating directly or indirectly with one or more other network nodes, as well as with a user device. In this regard, the communication circuitry 202 may comprise a Network Interface Circuit (NIC) (e.g., an ETHERNET or similar interface). In some embodiments, the network node 200 may be configured for wireless communications, and thus, communication circuitry 202 would comprise the radio frequency (RF) circuitry needed for transmitting and receiving signals over a wireless communication channel. In these cases, network node 200 may also be coupled to one or more antenna (not shown). In other embodiments, however, network node 200 is configured to communicate via a wire interface.

[0118] The processing circuitry 204 comprises one or more microprocessors, hardware, firmware, or a combination thereof that controls the overall operation of the network node 200. In accordance with the present disclosure, the processing circuitry 204 can be configured by software to perform one or more of the methods herein described including any of methods 10, 30, 40, 50, 70, 80, and 90 seen in Figures 1-6.

[0119] Memory 206 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuitry 204 for operation. Memory 206 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 206 stores a computer program 208 comprising executable instructions that configure the processing circuit 204 in the network node 200 to perform one or more of the methods herein described including any of the methods 10, 30, 40, 50, 70, 80, and 90 seen in Figures 1- 6. A computer program 208 in this regard may comprise one or more code modules corresponding to the means or units described above.

[0120] In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random-access memory (RAM). In some embodiments, computer program 308 for configuring the processing circuitry 304 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 308 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0121] Those skilled in the art will also appreciate that embodiments herein further include corresponding computer programs. A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

[0122] Embodiments of the present disclosure further include a carrier containing such a computer program 208. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0123] In this regard, embodiments herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform as described above.

[0124] Embodiments further include a computer program product comprising program code portions for performing the steps of any of the embodiments herein when the computer program product is executed by network node 200. This computer program product may be stored on a computer readable recording medium. Additional embodiments will now be described. At least some of these embodiments may be described as applicable in certain contexts and / or wireless network types for illustrative purposes, but the embodiments are similarly applicable in other contexts and / or wireless network types not explicitly described.

[0125] Figure 8 shows an example of a communication system 1100 in accordance with some embodiments.

[0126] In the example, the communication system 1100 includes a telecommunication network 1102 that includes an access network 1104, such as a radio access network (RAN), and a core network 1106, which includes one or more core network nodes 1108. The access network 1104 includes one or more access network nodes, such as network nodes 1110a and 1110b (one or more of which may be generally referred to as network nodes 1110), or any other similar 3rd Generation Partnership Project (3GPP) access nodes or non-3GPP access points. Moreover, as will be appreciated by those of skill in the art, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments, the telecommunication network 1102 includes one or more Open-RAN (ORAN) network nodes. An ORAN network node is a node in the telecommunication network 1102 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other nodes to implement one or more functionalities of any node in the telecommunication network 1102, including one or more network nodes 1110 and / or core network nodes 1108.

[0127] Examples of an ORAN network node include an open radio unit (O-RU), an open distributed unit (O-DU), an open central unit (O-CU), including an O-CU control plane (O-CU- CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time control application (e.g., xApp) or a non-real time control application (e.g., rApp), or any combination thereof (the adjective “open” designating support of an ORAN specification). The network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an A1 , F1 , W1 , E1 , E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN access node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment (described further below) in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an 0-2 interface defined by the O-RAN Alliance or comparable technologies. The network nodes 1110 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs 1112A, 1112B, 1112C, and 1112D (one or more of which may be generally referred to as UEs 1112) to the core network 1106 over one or more wireless connections.

[0128] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 1100 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 1100 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0129] The UEs 1112 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes 1110 and other communication devices. Similarly, the network nodes 1110 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs 1112 and / or with other network nodes or equipment in the telecommunication network 1102 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network 1102.

[0130] In the depicted example, the core network 1106 connects the network nodes 1110 to one or more hosts, such as host 1116. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 1106 includes one more core network nodes (e.g., core network node 1108) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 1108. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0131] The host 1116 may be under the ownership or control of a service provider other than an operator or provider of the access network 1104 and / or the telecommunication network 1102, and may be operated by the service provider or on behalf of the service provider. The host 1116 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0132] As a whole, the communication system 1100 of Figure 8 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low- power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0133] In some examples, the telecommunication network 1102 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 1102 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 1102. For example, the telecommunications network 1102 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC)ZMassive loT services to yet further UEs.

[0134] In some examples, the UEs 1112 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 1104 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 1104. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0135] In the example, the hub 1114 communicates with the access network 1104 to facilitate indirect communication between one or more UEs (e.g., UE 1112c and / or 1112D) and network nodes (e.g., network node 1110B). In some examples, the hub 1114 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 1114 may be a broadband router enabling access to the core network 1106 for the UEs. As another example, the hub 1114 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 1110, or by executable code, script, process, or other instructions in the hub 1114. As another example, the hub 1114 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 1114 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, the hub 1114 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 1114 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub 1114 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy loT devices.

[0136] The hub 1114 may have a constant / persistent or intermittent connection to the network node 1110B. The hub 1114 may also allow for a different communication scheme and / or schedule between the hub 1114 and UEs (e.g., UE 1112C and / or 1112D), and between the hub 1114 and the core network 1106. In other examples, the hub 1114 is connected to the core network 1106 and / or one or more UEs via a wired connection. Moreover, the hub 1114 may be configured to connect to an M2M service provider over the access network 1104 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 1110 while still connected via the hub 1114 via a wired or wireless connection. In some embodiments, the hub 1114 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 1110B. In other embodiments, the hub 1114 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node 1110B, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0137] Figure 9 shows a UE 1200 in accordance with some embodiments. As used herein, a UE refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle, vehicle-mounted or vehicle embedded / integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB- loT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0138] A UE may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle-to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0139] The UE 1200 includes processing circuitry 1202 that is operatively coupled via a bus 1204 to an input / output interface 1206, a power source 1208, a memory 1210, a communication interface 1212, and / or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in Figure 9. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0140] The processing circuitry 1202 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 1210. The processing circuitry 1202 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 1202 may include multiple central processing units (CPUs).

[0141] In the example, the input / output interface 1206 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE 1200. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device. In some embodiments, the power source 1208 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 1208 may further include power circuitry for delivering power from the power source 1208 itself, and / or an external power source, to the various parts of the UE 1200 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 1208. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 1208 to make the power suitable for the respective components of the UE 1200 to which power is supplied.

[0142] The memory 1210 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 1210 includes one or more application programs 1214, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 1216. The memory 1210 may store, for use by the UE 1200, any of a variety of various operating systems or combinations of operating systems.

[0143] The memory 1210 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUlCC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 1210 may allow the UE 1200 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 1210, which may be or comprise a device-readable storage medium.

[0144] The processing circuitry 1202 may be configured to communicate with an access network or other network using the communication interface 1212. The communication interface 1212 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 1222. The communication interface 1212 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter 1218 and / or a receiver 1220 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 1218 and receiver 1220 may be coupled to one or more antennas (e.g., antenna 1222) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0145] In the illustrated embodiment, communication functions of the communication interface 1212 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and / or standards, such as IEEE 802.11 , Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / internet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0146] Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface 1212, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0147] As another example, a UE comprises a mobile device, such as phone or tablet having a depth sensor, Augmented Reality (AR) glasses, a Mixed Reality (MR) device, a Virtual Reality (VR) device, an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0148] A UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a head-mounted display for Augmented Reality (AR) or Virtual Reality (VR), a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an loT device comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the UE 1200 shown in Figure 9.

[0149] As yet another specific example, in an loT scenario, a UE may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another UE and / or a network node. The UE may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-loT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0150] In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

[0151] Figure 10 shows a network node 1300 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a UE and / or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)), O-RAN nodes or components of an O-RAN node (e.g., O-RU, O-DU, O-CU).

[0152] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units, distributed units (e.g., in an O-RAN access node) and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0153] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0154] The network node 1300 includes a processing circuitry 1302, a memory 1304, a communication interface 1306, and a power source 1308. The network node 1300 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node 1300 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 1300 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 1304 for different RATs) and some components may be reused (e.g., a same antenna 1310 may be shared by different RATs). The network node 1300 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 1300, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 1300.

[0155] The processing circuitry 1302 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node 1300 components, such as the memory 1304, to provide network node 1300 functionality.

[0156] In some embodiments, the processing circuitry 1302 includes a system on a chip (SOC). In some embodiments, the processing circuitry 1302 includes one or more of radio frequency (RF) transceiver circuitry 1312 and baseband processing circuitry 1314. In some embodiments, the radio frequency (RF) transceiver circuitry 1312 and the baseband processing circuitry 1314 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 1312 and baseband processing circuitry 1314 may be on the same chip or set of chips, boards, or units.

[0157] The memory 1304 may comprise any form of volatile or non-volatile computer-readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry 1302. The memory 1304 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 1302 and utilized by the network node 1300. The memory 1304 may be used to store any calculations made by the processing circuitry 1302 and / or any data received via the communication interface 1306. In some embodiments, the processing circuitry 1302 and memory 1304 is integrated.

[0158] The communication interface 1306 is used in wired or wireless communication of signaling and / or data between a network node, access network, and / or UE. As illustrated, the communication interface 1306 comprises port(s) / terminal(s) 1316 to send and receive data, for example to and from a network over a wired connection. The communication interface 1306 also includes radio front-end circuitry 1318 that may be coupled to, or in certain embodiments a part of, the antenna 1310. Radio front-end circuitry 1318 comprises filters 1320 and amplifiers 1322. The radio front-end circuitry 1318 may be connected to an antenna 1310 and processing circuitry 1302. The radio front-end circuitry may be configured to condition signals communicated between antenna 1310 and processing circuitry 1302. The radio front-end circuitry 1318 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 1318 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 1320 and / or amplifiers 1322. The radio signal may then be transmitted via the antenna 1310. Similarly, when receiving data, the antenna 1310 may collect radio signals which are then converted into digital data by the radio front-end circuitry 1318. The digital data may be passed to the processing circuitry 1302. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0159] In certain alternative embodiments, the network node 1300 does not include separate radio front-end circuitry 1318, instead, the processing circuitry 1302 includes radio front-end circuitry and is connected to the antenna 1310. Similarly, in some embodiments, all or some of the RF transceiver circuitry 1312 is part of the communication interface 1306. In still other embodiments, the communication interface 1306 includes one or more ports or terminals 1316, the radio front-end circuitry 1318, and the RF transceiver circuitry 1312, as part of a radio unit (not shown), and the communication interface 1306 communicates with the baseband processing circuitry 1314, which is part of a digital unit (not shown).

[0160] The antenna 1310 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna 1310 may be coupled to the radio front-end circuitry 1318 and may be any type of antenna capable of transmitting and receiving data and / or signals wirelessly. In certain embodiments, the antenna 1310 is separate from the network node 1300 and connectable to the network node 1300 through an interface or port.

[0161] The antenna 1310, communication interface 1306, and / or the processing circuitry 1302 may be configured to perform any receiving operations and / or certain obtaining operations described herein as being performed by the network node. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, the antenna 1310, the communication interface 1306, and / or the processing circuitry 1302 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and / or signals may be transmitted to a UE, another network node and / or any other network equipment.

[0162] The power source 1308 provides power to the various components of network node 1300 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 1308 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 1300 with power for performing the functionality described herein. For example, the network node 1300 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 1308. As a further example, the power source 1308 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0163] Embodiments of the network node 1300 may include additional components beyond those shown in Figure 10 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the network node 1300 may include user interface equipment to allow input of information into the network node 1300 and to allow output of information from the network node 1300. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 1300.

[0164] Figure 11 is a block diagram of a host 1400, which may be an embodiment of the host 1116 of Figure 8, in accordance with various aspects described herein. As used herein, the host 1400 may be or comprise various combinations hardware and / or software, including a standalone server, a blade server, a cloud-implemented server, a distributed server, a virtual machine, container, or processing resources in a server farm. The host 1400 may provide one or more services to one or more UEs.

[0165] The host 1400 includes processing circuitry 1402 that is operatively coupled via a bus 1404 to an input / output interface 1406, a network interface 1408, a power source 1410, and a memory 1412. Other components may be included in other embodiments. Features of these components may be substantially similar to those described with respect to the devices of previous figures, such as Figures 18 and 19, such that the descriptions thereof are generally applicable to the corresponding components of host 1400.

[0166] The memory 1412 may include one or more computer programs including one or more host application programs 1414 and data 1416, which may include user data, e.g., data generated by a UE for the host 1400 or data generated by the host 1400 for a UE. Embodiments of the host 1400 may utilize only a subset or all of the components shown. The host application programs 1414 may be implemented in a container-based architecture and may provide support for video codecs (e.g., Versatile Video Coding (WC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), MPEG, VP9) and audio codecs (e.g., FLAC, Advanced Audio Coding (AAC), MPEG, G.711), including transcoding for multiple different classes, types, or implementations of UEs (e.g., handsets, desktop computers, wearable display systems, heads-up display systems). The host application programs 1414 may also provide for user authentication and licensing checks and may periodically report health, routes, and content availability to a central node, such as a device in or on the edge of a core network. Accordingly, the host 1400 may select and / or indicate a different host for over-the-top services for a UE. The host application programs 1414 may support various protocols, such as the HTTP Live Streaming (HLS) protocol, Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), Dynamic Adaptive Streaming over HTTP (MPEG-DASH), etc.

[0167] Figure 12 is a block diagram illustrating a virtualization environment 1500 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 1500 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 1500 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an 0-2 interface.

[0168] Applications 1502 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0169] Hardware 1504 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1506 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 1508A and 1508B (one or more of which may be generally referred to as VMs 1508), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 1506 may present a virtual operating platform that appears like networking hardware to the VMs 1508.

[0170] The VMs 1508 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 1506. Different embodiments of the instance of a virtual appliance 1502 may be implemented on one or more of VMs 1508, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0171] In the context of NFV, a VM 1508 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 1508, and that part of hardware 1504 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 1508 on top of the hardware 1504 and corresponds to the application 1502.

[0172] Hardware 1504 may be implemented in a standalone network node with generic or specific components. Hardware 1504 may implement some functions via virtualization. Alternatively, hardware 1504 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 1510, which, among others, oversees lifecycle management of applications 1502. In some embodiments, hardware 1504 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1512 which may alternatively be used for communication between hardware nodes and radio units.

[0173] Figure 13 shows a communication diagram of a host 1602 communicating via a network node 1604 with a UE 1606 over a partially wireless connection in accordance with some embodiments. Example implementations, in accordance with various embodiments, of the UE (such as a UE 1112A of Figure 8 and / or UE 1200 of Figure 9), network node (such as network node 1110A of Figure 8 and / or network node 1300 of Figure 10), and host (such as host 1116 of Figure 8 and / or host 1400 of Figure 11) discussed in the preceding paragraphs will now be described with reference to Figure 13.

[0174] Like host 1400, embodiments of host 1602 include hardware, such as a communication interface, processing circuitry, and memory. The host 1602 also includes software, which is stored in or accessible by the host 1602 and executable by the processing circuitry. The software includes a host application that may be operable to provide a service to a remote user, such as the UE 1606 connecting via an over-the-top (OTT) connection 1650 extending between the UE 1606 and host 1602. In providing the service to the remote user, a host application may provide user data which is transmitted using the OTT connection 1650.

[0175] The network node 1604 includes hardware enabling it to communicate with the host 1602 and UE 1606. The connection 1660 may be direct or pass through a core network (like core network 1106 of Figure 8) and / or one or more other intermediate networks, such as one or more public, private, or hosted networks. For example, an intermediate network may be a backbone network or the Internet.

[0176] The UE 1606 includes hardware and software, which is stored in or accessible by UE 1606 and executable by the UE’s processing circuitry. The software includes a client application, such as a web browser or operator-specific “app” that may be operable to provide a service to a human or non-human user via UE 1606 with the support of the host 1602. In the host 1602, an executing host application may communicate with the executing client application via the OTT connection 1650 terminating at the UE 1606 and host 1602. In providing the service to the user, the UE's client application may receive request data from the host's host application and provide user data in response to the request data. The OTT connection 1650 may transfer both the request data and the user data. The UE's client application may interact with the user to generate the user data that it provides to the host application through the OTT connection 1650.

[0177] The OTT connection 1650 may extend via a connection 1660 between the host 1602 and the network node 1604 and via a wireless connection 1670 between the network node 1604 and the UE 1606 to provide the connection between the host 1602 and the UE 1606. The connection 1660 and wireless connection 1670, over which the OTT connection 1650 may be provided, have been drawn abstractly to illustrate the communication between the host 1602 and the UE 1606 via the network node 1604, without explicit reference to any intermediary devices and the precise routing of messages via these devices.

[0178] As an example of transmitting data via the OTT connection 1650, in step 1608, the host 1602 provides user data, which may be performed by executing a host application. In some embodiments, the user data is associated with a particular human user interacting with the UE 1606. In other embodiments, the user data is associated with a UE 1606 that shares data with the host 1602 without explicit human interaction. In step 1610, the host 1602 initiates a transmission carrying the user data towards the UE 1606. The host 1602 may initiate the transmission responsive to a request transmitted by the UE 1606. The request may be caused by human interaction with the UE 1606 or by operation of the client application executing on the UE 1606. The transmission may pass via the network node 1604, in accordance with the teachings of the embodiments described throughout this disclosure. Accordingly, in step 1612, the network node 1604 transmits to the UE 1606 the user data that was carried in the transmission that the host 1602 initiated, in accordance with the teachings of the embodiments described throughout this disclosure. In step 1614, the UE 1606 receives the user data carried in the transmission, which may be performed by a client application executed on the UE 1606 associated with the host application executed by the host 1602.

[0179] In some examples, the UE 1606 executes a client application which provides user data to the host 1602. The user data may be provided in reaction or response to the data received from the host 1602. Accordingly, in step 1616, the UE 1606 may provide user data, which may be performed by executing the client application. In providing the user data, the client application may further consider user input received from the user via an input / output interface of the UE 1606. Regardless of the specific manner in which the user data was provided, the UE 1606 initiates, in step 1618, transmission of the user data towards the host 1602 via the network node 1604. In step 1620, in accordance with the teachings of the embodiments described throughout this disclosure, the network node 1604 receives user data from the UE 1606 and initiates transmission of the received user data towards the host 1602. In step 1622, the host 1602 receives the user data carried in the transmission initiated by the UE 1606.

[0180] One or more of the various embodiments improve the performance of OTT services provided to the UE 1606 using the OTT connection 1650, in which the wireless connection 1670 forms the last segment. More precisely, when provided as an OTT service, for example, the teachings of these embodiments provide the capability for network nodes to avoid making incorrect decisions based on low quality predicted pose data, thereby beneficially avoiding revoke / rollback procedures. In more detail, the results of an incorrect decision could be visible to the simulation engine for an extended period of time. This increases the chances that the simulation engine will continue to make incorrect decisions. Moreover, in at least some cases, the results of an incorrect decision may be too complex to revert or to rollback. Therefore, by configuring the network nodes to avoid making incorrect decisions, the present embodiments improve both the performance of the network and the experience of the end users.

[0181] In an example scenario, factory status information may be collected and analyzed by the host 1602. As another example, the host 1602 may process audio and video data which may have been retrieved from a UE for use in creating maps. As another example, the host 1602 may collect and analyze real-time data to assist in controlling vehicle congestion (e.g., controlling traffic lights). As another example, the host 1602 may store surveillance video uploaded by a UE. As another example, the host 1602 may store or control access to media content such as video, audio, VR or AR which it can broadcast, multicast or unicast to UEs. As other examples, the host 1602 may be used for energy pricing, remote control of non-time critical electrical load to balance power generation needs, location services, presentation services (such as compiling diagrams etc. from data collected from remote devices), or any other function of collecting, retrieving, storing, analyzing and / or transmitting data.

[0182] In some examples, a measurement procedure may be provided for the purpose of monitoring data rate, latency and other factors on which the one or more embodiments improve. There may further be an optional network functionality for reconfiguring the OTT connection 1650 between the host 1602 and UE 1606, in response to variations in the measurement results. The measurement procedure and / or the network functionality for reconfiguring the OTT connection may be implemented in software and hardware of the host 1602 and / or UE 1606. In some embodiments, sensors (not shown) may be deployed in or in association with other devices through which the OTT connection 1650 passes; the sensors may participate in the measurement procedure by supplying values of the monitored quantities exemplified above, or supplying values of other physical quantities from which software may compute or estimate the monitored quantities. The reconfiguring of the OTT connection 1650 may include message format, retransmission settings, preferred routing etc.; the reconfiguring need not directly alter the operation of the network node 1604. Such procedures and functionalities may be known and practiced in the art. In certain embodiments, measurements may involve proprietary UE signaling that facilitates measurements of throughput, propagation times, latency and the like, by the host 1602. The measurements may be implemented in that software causes messages to be transmitted, in particular empty or ‘dummy’ messages, using the OTT connection 1650 while monitoring propagation times, errors, etc.

[0183] Although the computing devices described herein (e.g., UEs, network nodes, hosts) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0184] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer- readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally. The present embodiments may, of course, be carried out in other ways than those specifically set forth herein without departing from characteristics described herein. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, and all changes coming within the meaning and equivalency range of the appended claims are intended to be embraced therein.

Claims

CLAIMS1 . A method (70), implemented in an object detector node (200) of a communications network, of processing Augmented Reality (AR) / Mixed Reality (MR) data, the method comprising: detecting (72) a pose of an object based on sensor data provided by one or more sensors; generating (74) pose output data for the pose, wherein the pose output data comprises: pose data indicating a detected position and a detected orientation of the object in space; and pose accuracy data indicating an accuracy of the pose data relative to one of: actual pose data that indicates an actual position and an actual orientation of the object in space; and previously detected pose data; and sending (78) the pose output data to one or more nodes in the communications network.

2. The method of claim 1 , wherein the pose accuracy data comprises a pose accuracy indicator indicating a type of network node as being the source of the pose data.

3. The method of any of claims 1-2, wherein the pose accuracy data comprises a Circular Error Probable (CEP) value representing whether the actual position and / or the actual orientation of the object is within a predetermined percentile of the detected position and / or the detected orientation of the object, respectively.

4. The method of any of claims 1-2, wherein the pose accuracy data comprises a confidence value indicating a percentage that the actual position of the object and the detected position of the object are within a predetermined distance of each other.

5. The method of any of claims 1-2, wherein the pose accuracy data comprises a percentage value indicating the accuracy of the pose data relative to previously detected pose data.

6. The method of any of claims 1-5, further comprising: encoding (76) information that specifies: an algorithm that was used by the object detector node to detect the object; and / or one or more parameters used by the algorithm to detect the object; and sending the information to the one or more nodes.

7. The method of any of claims 1-6, wherein the one or more nodes in the communications network comprise a pose predictor node configured to predict a future pose of the object.

8. The method of any of claims 1-7, wherein the one or more nodes in the communications network comprise a simulator node configured to implement a simulation engine.

9. An object detector node (200) in a communications network for processing Augmented Reality (AR) / Mixed Reality (MR) data, the object detector node being configured to: detect (72) a pose of an object based on sensor data provided by one or more sensors; generate (74) pose output data for the pose, wherein the pose output data comprises: pose data indicating a detected position and a detected orientation of the object in space; and pose accuracy data indicating an accuracy of the pose data relative to one of: actual pose data that indicates an actual position and an actual orientation of the object in space; and previously detected pose data; and send (78) the pose output data to one or more nodes in the communications network.

10. The object detector node of claim 9, wherein the object detector node is further configured to perform the method according to any one of claims 2-8.11 . An object detector node (200) in a communications network for processing AugmentedReality (AR) / Mixed Reality (MR) data, the object detector node comprising: processing circuitry (204); and memory (206) configured to store instructions (208) executable by the processing circuitry, whereby the processing circuitry is configured to: detect (72) a pose of an object based on sensor data provided by one or more sensors; generate (74) pose output data for the pose, wherein the pose output data comprises: pose data indicating a detected position and a detected orientation of the object in space; and pose accuracy data indicating an accuracy of the pose data relative to one of: actual pose data that indicates an actual position and an actual orientation of the object in space; and previously detected pose data; andsend (78) the pose output data to one or more nodes in the communications network.

12. The object detector node of claim 11 , whereby the processing circuitry is further configured to perform the method according to any one of claims 2-8.

13. A computer program (208) comprising instructions that, when executed on processing circuitry (204) of an object detector node (200) configured to process Augmented Reality (AR) / Mixed Reality (MR) data, causes the object detector node to perform the method according to any of claims 1-8.

14. A non-transitory computer-readable storage medium (206) comprising a computer program (208) stored thereon, the computer program comprising executable instructions that, when executed by processing circuitry (204) in an object detector node (200) configured to process Augmented Reality (AR) / Mixed Reality (MR) data, causes the object detector node to perform the method of any one of claims 1-8.

15. A method (80), implemented in an object predictor node (200) in a communications network, of processing Augmented Reality (AR) / Mixed Reality (MR) data, the method comprising: receiving (82), from an object detector node, pose output data for an object detected by the object detector node, the pose output data comprising: pose data indicating a detected position and a detected orientation of the object in space; and pose accuracy data indicating an accuracy of the pose data relative to one of: actual pose data that indicates an actual position and an actual orientation of the object in space; and previously detected pose data; generating (84), based on the pose output data, predicted pose output data comprising: predicted pose data indicating a predicted position and a predicted orientation of the object in space; and prediction accuracy data indicating an accuracy of the predicted pose data relative to one of: the actual pose data that indicates the actual position and the actual orientation of the object in space; and previously predicted pose data;sending (88) the predicted pose output data to one or more nodes in the communications network.

16. The method of claim 15, wherein the prediction accuracy data comprises a prediction accuracy indicator indicating a type of network node as being the source of the predicted pose data.

17. The method of any of claims 15-16, wherein the prediction accuracy data comprises a Circular Error Probable (CEP) value representing whether the actual position and / or the actual orientation of the object is within a predetermined percentile of the predicted position and / or the predicted orientation of the object, respectively.

18. The method of any of claims 15-18, wherein the prediction accuracy data comprises a confidence value indicating a percentage that the actual position of the object and the predicted position of the object are within a predetermined distance of each other.

19. The method of any of claims 15-18, wherein the prediction accuracy data comprises a percentage value indicating the accuracy of the predicted pose data relative to previously predicted pose data.

20. The method of any of claims 15-19, further comprising: encoding (86) information that specifies: an algorithm that was used by the object predictor node to determine the predicted position and / or the predicted orientation of the object; and / or one or more parameters used by the algorithm to determine the predicted position and / or the predicted orientation of the object; and sending the information to the one or more nodes in the communications network.21 . The method of any of claims 15-20, wherein the one or more nodes in the communications network comprise a simulator node configured to implement a simulation engine.

22. The method of any of the preceding claims, wherein the type of node is one of: an object detector; and an object predictor.

23. An object predictor node (200) in a communications network for processing Augmented Reality (AR) / Mixed Reality (MR) data, the object predictor node being configured to:receive (82), from an object detector node, pose output data for an object detected by the object detector node, the pose output data comprising: pose data indicating a detected position and a detected orientation of the object in space; and pose accuracy data indicating an accuracy of the pose data relative to one of: actual pose data that indicates an actual position and an actual orientation of the object in space; and previously detected pose data; generate (84), based on the pose output data, predicted pose output data comprising: predicted pose data indicating a predicted position and a predicted orientation of the object in space; and prediction accuracy data indicating an accuracy of the predicted pose data relative to one of: the actual pose data that indicates the actual position and the actual orientation of the object in space; and previously predicted pose data; send (88) the predicted pose output data to one or more nodes in the communications network.

24. The object predictor node of claim 23, wherein the object predictor node is further configured to perform the method according to any one of claims 16-22.

25. An object predictor node (200) in a communications network for processing Augmented Reality (AR) / Mixed Reality (MR) data, the object predictor node comprising: processing circuitry (204); and memory (206) configured to store instructions (208) executable by the processing circuitry, whereby the processing circuitry is configured to: receive (82), from an object detector node, pose output data for an object detected by the object detector node, the pose output data comprising: pose data indicating a detected position and a detected orientation of the object in space; and pose accuracy data indicating an accuracy of the pose data relative to one of: actual pose data that indicates an actual position and an actual orientation of the object in space; and previously detected pose data; generate (84), based on the pose output data, predicted pose output data comprising:predicted pose data indicating a predicted position and a predicted orientation of the object in space; and prediction accuracy data indicating an accuracy of the predicted pose data relative to one of: the actual pose data that indicates the actual position and the actual orientation of the object in space; and previously predicted pose data; send (88) the predicted pose output data to one or more nodes in the communications network.

26. The object predictor node of claim 25, whereby the processing circuitry is further configured to perform the method according to any one of claims 16-22.

27. A computer program (208) comprising instructions that, when executed on processing circuitry (204) of an object predictor node (200) configured to process Augmented Reality (AR) / Mixed Reality (MR) data, causes the object predictor node to perform the method according to any of claims 15-22.

28. A non-transitory computer-readable storage medium (206) comprising a computer program (208) stored thereon, the computer program comprising executable instructions that, when executed by processing circuitry (204) in an object predictor node configured to process Augmented Reality (AR) / Mixed Reality (MR) data, causes the object predictor node to perform the method of any one of claims 15-22.

29. A method (90), performed by a network simulator node (200) implementing a simulation engine in a communications network, of processing Augmented Reality (AR) / Mixed Reality (MR) data, the method comprising: receiving (92) pose data output from an object detector node and / or prediction data output from an object predictor node: wherein the pose data output comprises: pose data indicating a detected position and a detected orientation of the object in space; and pose accuracy data indicating an accuracy of the pose data relative to one of: actual pose data that indicates an actual position and an actual orientation of the object in space; and previously detected pose data; and wherein the predicted pose output data comprises:predicted pose data indicating a predicted position and a predicted orientation of the object in space; and prediction accuracy data indicating an accuracy of the predicted pose data relative to one of: the actual pose data that indicates the actual position and the actual orientation of the object in space; and previously predicted pose data; updating (94) a simulation component of a world state of the simulation engine based on the pose accuracy data and the prediction accuracy data; and sending (96) AR / MR data to a user for display based on the updated one or more simulation components of the world state.

30. The method of claim 29, further comprising determining (52) which of the one or more simulation components of the world state to update based on the pose accuracy data and the prediction accuracy data.31 . The method of any of claims 29-30, wherein updating a simulation component of a world state comprises updating an object pose of the object according to the pose data.

32. The method of any of claims 29-30, wherein updating a simulation component of a world state comprises updating an object pose of the object according to the predicted pose data responsive to determining that: the pose accuracy data does not meet or exceed a first predetermined threshold value; and the predicted accuracy data meets or exceeds a second predetermined threshold value.

33. The method of any of claims 29-30, wherein updating a simulation component of a world state comprises updating an object pose of the object according to the pose data responsive to determining that the pose accuracy data meets or exceeds a predetermine threshold value.

34. The method of any of claims 29-30, wherein updating a simulation component of a world state comprises updating an object pose of the object according to the predicted pose data responsive to determining that the predicted pose accuracy data meets or exceeds a predetermine threshold value.

35. The method of any of claims 29-34, further comprising delaying or postponing one or more decisions by the simulation engine based on an accuracy of the pose data.

36. The method of any of claims 29-35, wherein the previously detected pose data indicates a previously detected position and a previously detected orientation of the object in space and is stored as being the most accurate detected pose data.

37. A network simulator node (200) in a communications network configured to implement a simulation engine for processing Augmented Reality (AR) / Mixed Reality (MR) data, the network simulator node being configured to: receive (92) pose data output from an object detector node and prediction data output from an object predictor node: wherein the pose data output comprises: pose data indicating a detected position and a detected orientation of the object in space; and pose accuracy data indicating an accuracy of the pose data relative to one of: actual pose data that indicates an actual position and an actual orientation of the object in space; and previously detected pose data; and wherein the predicted pose output data comprises: predicted pose data indicating a predicted position and a predicted orientation of the object in space; and prediction accuracy data indicating an accuracy of the predicted pose data relative to one of: the actual pose data that indicates the actual position and the actual orientation of the object in space; and previously predicted pose data; update (94) a simulation component of a world state of the simulation engine based on the pose accuracy data and the prediction accuracy data; and send (96) AR / MR data to a user device for display based on the updated one or more simulation components of the world state.

38. The network simulator node of claim 37, wherein the network simulator node is further configured to perform the method according to any one of claims 30-36.

39. A network simulator node (200) in a communications network for processing Augmented Reality (AR) / Mixed Reality (MR) data, the network simulator node comprising: processing circuitry (204); and memory (206) configured to store instructions (208) executable by the processing circuitry, whereby the processing circuitry is configured to:receive (92) pose data output from an object detector node and prediction data output from an object predictor node: wherein the pose data output comprises: pose data indicating a detected position and a detected orientation of the object in space; and pose accuracy data indicating an accuracy of the pose data relative to one of: actual pose data that indicates an actual position and an actual orientation of the object in space; and previously detected pose data; and wherein the predicted pose output data comprises: predicted pose data indicating a predicted position and a predicted orientation of the object in space; and prediction accuracy data indicating an accuracy of the predicted pose data relative to one of: the actual pose data that indicates the actual position and the actual orientation of the object in space; and previously predicted pose data; update (94) a simulation component of a world state of the simulation engine based on the pose accuracy data and the prediction accuracy data; and send (96) AR / MR data to a user device for display based on the updated one or more simulation components of the world state.

40. The network simulator node of claim 39, whereby the processing circuitry is further configured to perform the method according to any one of claims 30-36.

41. A computer program (208) comprising instructions that, when executed on processing circuitry (204) of a network simulator node (200) configured to process Augmented Reality (AR) / Mixed Reality (MR) data, causes the network simulator node to perform the method according to any of claims 29-36.

42. A carrier containing the computer program of any one of claims 13, 27, and 41 , wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

43. A non-transitory computer-readable storage medium (206) comprising a computer program (208) stored thereon, the computer program comprising executable instructions that, whenexecuted by processing circuitry (204) in a network simulator node (200) configured to process Augmented Reality (AR) / Mixed Reality (MR) data, causes the network simulator node to perform the method of any one of claims 31-38.

44. A system (30, 40, 50) configured to process Augmented Reality (AR) / Mixed Reality (MR) data, the system comprising: an object detector node configured to: generate (74) pose output data for a pose of an object in space, wherein the pose output data comprises: pose data indicating a detected position and a detected orientation of the object in space; and pose accuracy data indicating an accuracy of the pose data relative to one of: actual pose data that indicates an actual position and an actual orientation of the object in space; and previously detected pose data; an object predictor node configured to: generate (84), based on the pose output data, predicted pose output data comprising: predicted pose data indicating a predicted position and a predicted orientation of the object in space; and prediction accuracy data indicating an accuracy of the predicted pose data relative to one of: the actual pose data that indicates the actual position and the actual orientation of the object in space; and previously predicted pose data; and a network simulator node configured to: update (94) a simulation component of a world state of a simulation engine based on the pose accuracy data and the prediction accuracy data; and send (96) AR / MR data to a user device for display based on the updated one or more simulation components of the world state.

Images