Device for providing synchronous input

Abstract

The present invention relates to an apparatus for providing a synchronization input to a third device, the apparatus comprising: a. A memory for storing communication parameters shared between the third device and a first device and / or between the third device and a second device, b. A communication unit for receiving a first communication stream from the first device, and / or receiving a second communication stream from the second device, a second communication flow is received from a second device, where the communication flow is synchronized based on the communication parameters.

Description

Technical Field

[0001] The present invention relates to communication systems requiring low latency communication between remote users. The present invention may be applicable to (but not limited to) next generation real-time communication systems or metaverse implementations, which may be defined as virtual reality spaces in which users can interact with computer-generated environments and other users. Background Art

[0002] Within the 3GPP Technical Specification Group Services and System Aspects (TSG SA), the main goal of 3GPP TSG SA WG1 (SA1) is to consider and study new and enhanced services, features and capabilities for 5G systems, and to identify any corresponding Stage 1 requirements to be met by the 3GPP specifications. These service requirements are documented in the standard specifications under the responsibility of SA1. A related study is TR22.847, "Study on supporting tactile and multi-modality communication services (TAMMCS)". This study includes eight use cases and related requirements for the so-called "Tactile Internet" (TI).

[0003] The International Telecommunication Union (ITU) defines TI as an Internet network that combines ultra-low latency with extremely high availability, reliability and security. Mobile Internet allows the exchange of data and multimedia content on the move. The next step is the Internet of Things (IoT) for interconnecting smart devices. TI is the next evolution of IoT that will enable real-time control. It will add new dimensions to human-machine interaction by enabling tactile and haptic sensations, and at the same time revolutionize the interaction of machines. TI will enable people and machines to interact with their environment in real time while moving and within a certain spatial communication range.

[0004] IEEE Publication P1918.1, "Tactile Internet: Application Scenarios, Definitions and Terminology, Architecture, Functions, and Technical Assumptions," requires that cellular 5G communication systems should support a mechanism to assist in synchronization between multiple streams (e.g., tactile, audio, and video) of a multimodal communication session to avoid negative impacts on the user experience. In addition, the 5G system should be able to support interaction with applications on user equipment (UE) or data stream grouping information within a tactile and multimodal communication service, and support units for applying third-party provided policies for streams associated with an application. A policy can contain a set of UEs and data streams, expected quality of service (QoS) treatment and associated triggering events, and other coordination information.

[0005] Figure 3 The scenario solved by the present invention is depicted. Two persons A and B wish to interact in a virtual reality or real-time communication application. To this end, persons A and B have corresponding rendering devices, such as VR devices and corresponding sensor devices. Persons A and B are separated by a distance d.

[0006] There are two main problems with this situation:

[0007] First, if the virtual reality application requires a maximum 1 ms delay, the maximum distance between person A and person B is d = 300 km, since information propagates at 300.000 km / s.

[0008] Second, because high-quality data representation is required, the sensor device needs to sample a high-quality representation of one person, which will be sent to the rendering device of the other person. This results in high data rate requirements.

[0009] In addition, more than two people may be involved in a virtual reality, where different people may be in different locations. For example, assume that users U0, U1, and U2 interact with each other at different locations L0, L1, and L2. User U0 receives data from users U1 and U2. When the data arrives at U0, the data streams generated by U1 and U2 and the data streams of the UE local to U0 are required to be synchronized.

[0010] Therefore, the third problem is how to synchronize the data streams originating from UEs at different locations. Summary of the invention

[0011] The object of the present invention is to alleviate the above mentioned problems.

[0012] Another object of the present invention is to enable virtual reality interaction between people at remote locations and reduce communication overhead.

[0013] According to the general definition of the present invention, a first person interacts with a second person in a virtual reality through a prediction model of the second person located near the first person. The prediction model predicts the state at time t based on sensor inputs sampled at time t(0)=td*c, t(1)=td*cT,…, t(N-1)=td*c-(N-1)*T, where c is the speed of light,

[0014] The method may allow one or more of the following:

[0015] First, low latency: Instead of direct interaction, person A(B) interacts with a predictive model of person B(A). The model of person B(A) predicts the actions of person B(A) based on the input collected by sensor device B. The predictive model of person B(A) is used in the rendering device A(B).

[0016] A model of a person can be built when the person joins a virtual reality, and can be deployed to a suitable location when the person wants to interact with another person'. The suitable location must be close to the person'.

[0017] Second, data compression: This is achieved by: a) downloading a complex model of the person at a suitable location during initialization; b) limiting the sensor data that needs to be sampled for a suitable prediction model that leads to a realistic representation of the person;

[0018] Another general definition of the invention proposes synchronization of (predicted) communication flows from different user equipment at different locations by, for example:

[0019] First, determine the communication parameters (e.g., latency) between users;

[0020] Second, synchronize the streams based on the communication parameters;

[0021] Third, this synchronization is applied to the use of predictive models.

[0022] According to a first aspect of the present invention, a device for providing a synchronization input to a third device is provided, the device comprising:

[0023] a memory for storing communication parameters shared between the third device and the first device and between the third device and the second device,

[0024] b. a communication unit for receiving a first communication stream from a first device and a second communication stream from a second device,

[0025] The first communication flow and the second communication flow are synchronized based on the communication parameter.

[0026] According to a second aspect of the present invention, a system is proposed, comprising at least one communication device according to the first aspect of the present invention and at least one remote second device for transmitting a received communication stream.

[0027] According to a third aspect of the present invention, a device for providing a derived prediction model of a first device to a third device is provided. The device comprises:

[0028] a. a storage unit for storing a general prediction model of the first device,

[0029] b. a communication unit for receiving a communication parameter characteristic of a communication link between the first device and the third device;

[0030] c. a computing unit capable of determining a derived prediction model of the first device,

[0031] The derived prediction model is obtained based on the general prediction model and the received communication parameters.

[0032] According to another general definition of the present invention, there is provided an apparatus for providing a synchronization input to a third device, the apparatus comprising

[0033] a memory for storing communication parameters shared between the third device and the first device and / or between the third device and the second device,

[0034] b. a communication unit for receiving a first communication stream from a first device and / or receiving a second communication stream from a second device,

[0035] The communication flows are synchronized based on the communication parameters.

[0036] In a first variant of this general definition of the invention, the communication parameters include:

[0037] - the distance between the third device and the first device and / or the distance between the third device and the second device, or

[0038] - the latency from the first device to the third device and / or the latency from the second device to the third device, or

[0039] - Other communication parameters shared between the first device and the third device and / or between the second device and the third device.

[0040] In a second variation, the apparatus further comprises a computing unit for executing a prediction model, the prediction model taking as input the communication parameters between the first device and the third device to predict a control input for the third device, wherein the control input comprises the predicted communication parameters between the first device and the third device.

[0041] In a third variant, the apparatus comprises a computation unit for executing a prediction model taking as input at least a communication flow of the first device to predict a control input for a third device.

[0042] In a fourth variation, the prediction model is at least one of:

[0043] - a model derived from a common prediction model based on parameters shared between at least the first device and the third device,

[0044] -Generate models.

[0045] In a fifth variation, the communication parameters are obtained by running a protocol with the first device.

[0046] In a sixth variation, the communication parameters are configured by a management device.

[0047] In a seventh variation, the communication parameter may be at least one of the following:

[0048] - a delay between the third device and the first device and / or the second device,

[0049] -QoS,

[0050] - a distance between the third device and the first device and / or the second device,

[0051] - the computational requirements for handling communications,

[0052] - computing power to handle communications,

[0053] - memory requirements for handling communications,

[0054] - memory capacity for handling communications,

[0055] - available bitrate,

[0056] - the number of communicating parties,

[0057] - a (relative) position relative to the third device and the first device and / or the second device,

[0058] - a (relative) speed relative to the third device and the first device and / or the second device,

[0059] - a (relative) acceleration relative to the third device and the first device and / or the second device,

[0060] - a (relative) rotation relative to the third device and the first device and / or the second device.

[0061] Under this other general definition, the invention is also directed to a system comprising at least one third device and at least one remote first device, the third device comprising means as defined in the previous definition and variants thereof, the remote first device sending a communication stream for reception at the third device.

[0062] Still according to the other general definition, a method for providing a synchronization input to a third device is also proposed, the method

[0063] a. storing in a memory communication parameters shared between the third device and the first device, and / or between the third device and the second device;

[0064] b. receiving a first communication stream from a first device through a communication unit, and / or a second communication stream from a second device;

[0065] c. Synchronize the communication flow based on the communication parameters.

[0066] Still under this other general definition, there is also proposed an apparatus for using a prediction model of a first device, the apparatus comprising

[0067] a. a storage unit for storing a prediction model of a first device;

[0068] b. a communication unit for acquiring communication parameter characteristics of a communication link between the first device and the third device;

[0069] c. a computing unit capable of using the prediction model of the first device,

[0070] The output of the prediction model is obtained based on the prediction model and communication parameters.

[0071] According to a first variant, the output of the prediction model allows a synchronous communication flow between the first device and the third device.

[0072] According to a second variant, the output of the prediction model is a derived prediction model, and wherein the required number of input parameters in the derived prediction model is smaller than the required number of input parameters of the prediction model.

[0073] Still under this other general definition, a method is also proposed using a prediction model of a first device, the method being suitable for

[0074] a. storing the prediction model of the first device;

[0075] b. obtaining communication parameter characteristics of the communication link between the first device and the third device;

[0076] c. Using the prediction model of the first device, an output based on the prediction model is obtained based on the prediction model and the communication parameters.

[0077] It shall be understood that a preferred embodiment of the invention can also be any combination of the dependent claims or the above-described embodiments with the respective independent claim.

[0078] It will be understood that some or all of the aspects previously described may be implemented by a computer program comprising instructions which, once executed on a computer, enable the implementation of the methods covered by the present invention.

[0079] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1A and Figure 1B A block diagram schematically illustrates an alternative network architecture for virtual reality implementation in a cellular network;

[0081] Figure 2 schematically illustrates a state diagram representing the operation of an exemplary system for implementing a metaverse application;

[0082] Figure 3 An exemplary virtual reality implementation is schematically illustrated;

[0083] Figure 4 A virtual reality system according to an embodiment of the present invention is shown;

[0084] Figure 5 It means a virtual reality system implemented in a network;

[0085] Figure 6 schematically illustrates a system according to another embodiment of the present invention;

[0086] Figure 7 schematically illustrates a system according to another embodiment of the present invention;

[0087] Figure 8 Schematically represents a variation of an embodiment of the present invention;

[0088] 9A-D schematically illustrate a system according to another embodiment of the present invention;

[0089] Fig.10 schematically illustrates a system according to another embodiment of the present invention;

[0090] Fig.11 schematically represents a system in which the foregoing embodiments may be implemented; and

[0091] Fig.12 Exemplary use cases for application of the embodiments are shown. DETAILED DESCRIPTION

[0092] Embodiments of the present invention are now described based on a cellular communication network environment such as 5G. However, the present invention may also be used in conjunction with other wireless technologies in which TI or virtual reality applications are provided or introduced. The present invention may also be applied to other applications such as video streaming services, video broadcast services, or data storage.

[0093] Throughout this disclosure, the abbreviation "gNB" (5G terminology) or "BS" (Base Station) is intended to represent a wireless access device, such as a cellular base station or WiFi access point or ultra-wideband (UWB) personal area network (PAN) coordinator. A gNB may include a centralized control plane unit (gNB-CU-CP), multiple centralized user plane units (gNB-CU-UP) and / or multiple distributed units (gNB-DU). A gNB is part of the Radio Access Network (RAN), which provides an interface to functions in the Core Network (CN). The RAN is part of a wireless communication network. It implements the Radio Access Technology (RAT). Conceptually, it resides between communication devices such as mobile phones, computers or any remotely controlled machines, and provides a connection to their CN. The CN is the core part of the communication network, which provides many services to interconnected customers via the RAN. More specifically, it directs the flow of communication through the communication network and possibly other networks.

[0094] In addition, the terms "base station" (BS) and "network" may be used as synonyms in the present disclosure. This means that, for example, when it is written that "the network" performs a certain operation, it may be performed by a CN function of a wireless communication network, or by one or more base stations that are part of such a wireless communication network, and vice versa. It may also mean that part of the function is performed by the CN function of the wireless communication network, and part of the function is performed by the base station.

[0095] Furthermore, the term "virtual reality" is understood to refer to a persistent shared collection of interactable spaces where users can interact with each other alongside mutually perceived virtual features (i.e., augmented reality (AR)), or where those spaces consist entirely of virtual features (i.e., virtual reality (VR)). VR and AR may generally be referred to as "mixed reality" (MR).

[0096] Additionally, the term "data" is understood to refer to a representation of information according to a known or agreed format to be stored, transmitted, or otherwise processed. The information may specifically include one or more channels of audio, video, image, tactile, motion, or other forms of multimedia information that may be synchronized. Such multimedia information may be derived from sensors (e.g., microphones, cameras, motion detectors, etc.), or may be partially or fully synthetic (e.g., live actors in front of a synthetic background).

[0097] The term "data object" is understood to mean one or more sets of data according to the above definition, optionally accompanied by one or more data descriptors that provide additional semantic information about the data that affects how the data should be processed at the transmitter and receiver. Data descriptors can be used to describe, for example, how the data is classified by the transmitter and how it should be rendered by the receiver. As an example, data used to represent an image or video sequence can be decomposed into a set of data objects that together describe the complete image or video and can be substantially independent of other data objects and processed individually (e.g., compressed) in a manner optimal for the objects and their semantic context.

[0098] Furthermore, the term "data object classification" is understood to refer to a process in which data is divided or segmented into a plurality of data objects. For example, an image may be divided into a plurality of parts, such as a forest in the background and a person in the foreground (e.g., as later combined with Figure 8 exemplified). Data object classification criteria are used to classify data objects. In the present disclosure, such criteria may include at least one of a measure of the semantic content of the data object, a context of the data object, a category of compression techniques that is most suitable for retaining sufficient semantic content for a given context, etc.

[0099] Furthermore, "compression techniques" are understood to mean methods of reducing the size of data so that its transmission or storage is more efficient. For example, a method of removing redundant data or data that is considered semantically insignificant to the end user and encoding the remaining data efficiently, making it possible to reconstruct a true or semantically close to true representation of the original data.

[0100] Furthermore, a "compression or reconstruction model" is understood to refer to a repository of tools and data objects that can be used to assist in data compression and reconstruction. For example, a model may include algorithms used in the analysis and compression of data objects, or may include data objects that can be used as a basis for generating compression techniques. Advantageously, a model may be shared or owned by a transmitter and a receiver, and / or may be updated or optimized depending on the semantic content of the data being transmitted.

[0101] Note that throughout this disclosure, only those blocks, components and / or devices related to the proposed data distribution functionality are shown in the accompanying drawings. For the sake of brevity, other blocks are omitted. In addition, blocks represented by the same reference numerals are intended to have the same or at least similar functions, so that their functions will not be described later.

[0102] Figure 1A and 1BThe network architecture (e.g., IEEE P1918.1 architecture) considered for implementing virtual reality is schematically shown. The architecture includes an actuator gateway (AG), an actuator node (AN), a controller node (CN), a control plane entity (CPE), a gateway node (GN, where GNC corresponds to GN and CN), a human-system interface node (HN), a network controller (NC), a sensor / actuator (S / A), a computing and storage entity (SE), a sensor gateway (SG), a sensor node (SN), a tactile device (TD), a tactile edge (TE), a tactile service manager (TSM), a user plane entity (UPE), an access interface (A), a first tactile interface Ta (TD to TD communication), a second tactile interface Tb (TD to GNC communication), an open interface (O), a service interface (S), a network side (N), a network domain (ND), a bidirectional information exchange (BIE), an external application service provider (EASP), and a dedicated low latency network (LLNW).

[0103] Figure 1A and Figure 1B The architecture provides an overall communication architecture defined in a generic way that can run on any network (including 5G) / on any network. They cover various modes of interconnected network domains between two TEs (TE A, TE B). Each TE includes one or more TDs, where the TD in TE A transmits (e.g., tactile / tactile) information with the TD in TE B through the ND to meet the requirements of a given TI use case. The ND can be a shared wireless network (e.g., 5G radio access and core network), a shared wired network (e.g., Internet core network), a dedicated wireless network (e.g., point-to-point microwave or millimeter wave link), or a dedicated wired network (e.g., point-to-point leased line or fiber optic link). Each TD can support one or more of the functions of sensing, actuation, tactile feedback, or control via one or more corresponding entities. The S or A entity refers to a device that performs sensing or actuation functions, respectively, without a networking module. SN or AN refers to a device that performs sensing or actuation functions, respectively, using an air interface network connection module. In order to connect S to SN or A to AN, SG or AG entities should be used, respectively. These gateways provide a common interface for connecting to third-party sensing and actuation devices and another interface for connecting to SNs and ANs. TDs can also be used as HNs, which can convert human input into tactile outputs, or as CNs, which run control algorithms for the operation of the system that utilizes the necessary network connection modules to handle SNs and ANs.

[0104] The GN is an entity with enhanced networking capabilities that resides at the interface between the TE and the ND and is primarily responsible for user plane data forwarding. The GN is accompanied by the NC, which is responsible for control plane processing, including intelligence for admission and congestion control, service provisioning, resource management and optimization, and connection management in order to achieve the required QoS for TI sessions. Depending on the network design and configuration, the GN and CN (together labeled GNC) can be located on the TE side (e.g. Figure 1A as shown) or ND side (as shown Figure 1B The GNC is a central node as it facilitates interoperability with a variety of possible network domain options, which is necessary for compatibility with other emerging standards such as the 3GPP 5G NR specification. Allowing the GNC to reside in the ND, for example under 5G, is intended to support the option of absorbing its functionality into the management and orchestration functions already there. Figure 1A and 1B In FIG, the ND is shown as consisting of radio access points or base stations logically connected to the CPEs and UPEs in the network core.

[0105] The user in the region of interest (ROI) is surrounded by a set of TDs linked to the TE. The TD may include a rendering actuator and / or a sensor. The rendering actuator has the task of creating a virtual reality environment around the user, and may be a VR glasses, a 3D television (TV), a holographic device, etc. The sensor TD is a device responsible for capturing the user's movements and / or environment, and may include a camera, an audio device (such as a microphone), a tactile sensor, etc. Typically, a TD may be a UE in terms of a 5G system.

[0106] The TD in the ROI can be connected to the user's TE, for example, by wired or wireless means. In the wireless case, the UE can be connected to a base station such as a 5G gNB or a WiFi access point. The networking infrastructure and computing resources of the TE are co-located in the ROI or located in a near-edge server (less than the maximum edge distance) to ensure fast response.

[0107] To help implement the following embodiments, at least one of three communication functions may be introduced. First, latency-based stream synchronization (LBFS) may be provided, which is a function that may be run in a device in the TE. It may also be deployed in a receiving TD that is capable of determining communication parameters with a sending TD (or TS) and synchronizing communication streams based on those communication parameters, especially the relative latency between TDs (or TEs). Second, the edge application at the TE may be configured to run a configurable prediction model (LDCPM) related to latency of the environment / people in a virtual reality session in different TEs. Third, a model management and configuration function may be provided that is capable of registering a generic model of ROI and / or devices and / or people in the TE, storing it in a database, and deploying a reconfigured LDCPM when determining communication parameters.

[0108] Figure 1A and Figure 1B Another pioneering node in the architecture is SE, which provides computing and storage resources for improving the performance of TE and meeting the latency and reliability requirements of E2E communication. SE will run advanced algorithms, among other items, using AI technology to unload processing operations (e.g., tactile rendering, motion trajectory prediction and sensory compensation) that cannot be completed in TD due to being too resource and / or energy intensive. The goal is to achieve the use of predictive analysis to perceive real-time connections, while overcoming the challenges and uncertainties along the path between the source TD and the destination TD, dynamically estimating the changes in network load and rate over time to optimize resource utilization, and allowing the sharing of learned experience about the environment between different TDs. On the other hand, SE will also provide intelligent caching capabilities, which may be very influential in reducing E2E business load and therefore in reducing data transmission delays. SE can reside locally in TE to enhance the response rate of requests from TD or GNC, and / or it can reside remotely in the cloud while providing services to TE and ND. In addition, SE can be centralized or distributed. Each of these options has its own advantages and disadvantages in terms of latency, reliability, capability, cost and practical feasibility. The communication between two TEs can be unidirectional or bidirectional, can be based on a client-server model or a peer-to-peer model, and can belong to any of the above use cases with their corresponding reliability and latency requirements. To this end, the TSM plays a key role in defining the characteristics and requirements of the service between the two TEs and in propagating this information to key nodes in the TE and ND. The TSM will also support functions such as registration and authentication, and will provide an interface to the EASP of the TI.

[0109] The A interface provides a connection between TE and ND. It is the main reference point for the exchange of user plane and control plane information between ND and TE. Depending on the architecture design, the A interface can be between TD and ND or between GNC and ND. In addition, the T interface provides a connection between entities within the TE. It is the main reference point for the exchange of control plane information between the user plane and the entities in the TE. The T interface is divided into two sub-interfaces Ta and Tb to support different modes of TD connectivity, whereby the Ta interface is used for TD to TD communication, and when the GNC resides in the TE, the Tb interface is used for TD to GNC communication. In addition, the O interface provides connectivity between any architectural entity and the SE, and the S interface provides connectivity between the TSM and the GNC. The S interface carries control plane information. Finally, the N interface refers to any interface that provides internal connectivity between ND entities. This is usually covered as part of the network domain standard and may include sub-interfaces for both user plane and control plane entities.

[0110] Two broad categories of tactile information can be realized, namely tactile or kinesthetic, which can be combined. Tactile information refers to the perception of information such as surface texture, friction and temperature by various mechanical stimuli receptors in the human skin. Kinesthetic information refers to information sensed by the human skeleton, muscles and tendons, such as force, torque, posture and speed.

[0111] The first differentiating aspect of TI and related standards compared to 5G Ultra-Reliable Low Latency Communications (URLLC, ITU-R M.2083) relates to the fact that, due to the propagation time of 1 ms, TI must be developed in a way that its requirements can be achieved over distances longer than the 150 km round trip (or 100 km in fiber) separation. This capability can be achieved through network-side support functions built into the TI architecture, as envisioned through the standards work in IEEE 1918.1. These functions can, for example, use artificial intelligence (AI) methods to model the remote environment, and in some cases can also be partially or fully present at the TI end device (i.e., the client of the TI / tactile information).

[0112] The second distinguishing aspect relates to the fact that TI leads to applications with the following: unique characteristics implied by the application, and it is expected that the application can be deployed as an overlay network on top of a network or a combination of networks. It is not intended to be applied only as an underlying communication unit in the context of 5G URLLC.

[0113] In addition, Figure 1A and Figure 1BIn the above architecture, data streams (e.g., haptic feedback) must also be synchronized, and users expect that they should "feel" or "experience" visually depicted events as they occur, regardless of whether the events are heard. Therefore, synchronization of audio, video, and haptic data becomes very critical. This may be accomplished concomitantly through receiver buffering, thereby completely eliminating the challenges of the communication network in achieving the required latency (e.g., jitter).

[0114] To meet stringent end-to-end (E2E) QoS requirements, the architecture should also provide advanced operations and management capabilities, such as lightweight signaling protocols, distributed computing and caching with predictive analytics, intelligent adaptation to load and network conditions, and integration with external application service providers (ASPs).

[0115] Therefore, reliability, latency, and scalability can be considered key performance indicators (KPIs), where omnidirectional (fast "latch" of TD to TI infrastructure), self-organizing (minimal maintenance of TI network domain), and hybrid (scalable but minimal maintenance of TI docking devices) can be considered the three main approaches for bootstrapping TI services and instantiating architectures. TI design is inherently built on the concept of edge-specified operational settings and core-managed E2E usage. That is, the TD at the edge will state its communication and operational parameters (e.g., expected latency and reliability) and communicate them to the TI architecture, which will then invest the required resources in bootstrapping settings and E2E communications to meet such requirements.

[0116] Figure 2 A state diagram representing the operating states of an overall operating finite state machine for implementing a metaverse application is schematically shown.

[0117] The TD device may start with a registration phase (REG), which is defined as the action of establishing communication with the TI architecture. Under the omnidirectional TI paradigm, registration will occur at the GNC, potentially including TI components from the ND, such as the TSM. The selected application may provide a user interface for the user / ROI to register, for example, in the TSM. Registration may be achieved through the application itself or through the TSM, and may include registering the TD that the user / ROI has. It may involve registering the TD as part of the communication infrastructure, for example, registering as part of the 5G System (5GS) to access functionality provided by the 5GS, such as Quality of Service (QoS), low latency, edge computing, or synchronization of communication flows.

[0118] During registration, the TSM may assign a TE to a user and / or ROI and / or TD that is close to or suitable for its communication parameters. In some cases, a sensor TD may generate output for being fed to a TD for rendering in the same ROI.

[0119] The "latch" point of the TD for initiating registration may be referred to as the TI anchor. At this stage, the TD may probe the TI architecture to invoke E2E communication and may not perform any other functions other than latching to the TI architecture. In the ad hoc and hybrid models, this step may involve the TSM, potentially via the GNC in the former, to establish registration.

[0120] The next state depends on the type of TD. If it is a low-end SN / AN, the TD may have a designated "parent" nearby, and the TD may need to associate with the "parent" first (ASS). Thereafter, the parent TI node can ensure reliable operation and assist in connection establishment and error recovery. If the TD device operates independently, this will be an optional step. Some mission-critical TDs as well as new mission-critical TDs may need to be authenticated (AUT) without a parent (Ap) before being allowed to join / start a TI session (SS).

[0121] Another stage may be an optional state, in which the TD (NATD) can communicate with the authentication agent in the TI infrastructure to perform authentication. TSM may be an entity that may perform this task with the help of SE when needed or in the case of a large amount of business. Then, TD can start its E2E control synchronization (Ctrl Sync), in which TD can detect and establish a link to the end TE. In this state, TD may not be allowed to transmit operational data, but may focus on relay connection establishment and maintenance parameters. This may include setting parameters for interfaces along the E2E path, which can help ND select the optimal path throughout the network to deliver the requested connection parameters. This state includes the path establishment and routing selection phase of TI operation. It may typically involve multiple layers of the TI architecture, which can communicate to ensure that the path that meets the minimum requirements set in the "Setup" message is indeed available and reserved.

[0122] If the TD participating in the TI session is a tactile node (HN) targeting tactile communication, the next state may include specific communication and establishment of tactile-specific information still before the actual data communication. This state involves deciding the codec, session parameters, and message format specific to the current TI session. Although different use cases may require different tactile exchange frequencies, it is expected that each tactile communication will start with a tactile synchronization state (H-Sync) to establish initial parameters. Future changes to the codec and other tactile parameters can then be processed as data communications in the "operational" state (OP). This ensures that all tactile communications will implement the initial settings, regardless of future updates to the parameters that may be included in the operational data payload.

[0123] All TD components may then transition to the Operational state. In this state, the E2E path has been established, all connection establishment requirements have been met, and the TE is ready to exchange TI information. During operation in this state, one TD may detect an intermittent network error (ERR), in which case the TD may transition to a "recovery" mode (REC) where the specified protocol may take over error checking and potential correction mechanisms to attempt to reestablish reliable communications. If the error proves to be intermittent and is resolved, the TD may transition back to the Operational state. If for any reason the error persists, the TD may transition back to Control Synchronization and rediscover whether the E2E path is indeed available under the operational requirements set by the edge user.

[0124] Finally, once the TI operation is successfully completed, the TD may transition to the "Termination" phase (TERM), where all resources previously dedicated to the TD may be released back to the TI management plane. If originally handled by the NC, the resources are returned to it. Most typically, TSM may involve the provisioning of TI resources.

[0125] Figure 3 Schematically showing two persons P A and P B An exemplary virtual reality scene in which two persons are willing to interact in virtual reality. To this end, two persons have corresponding rendering devices RD A and RD B, for example, virtual reality (VR) devices and corresponding sensor devices SD A and SD B. Two persons P A and P B Separated by a distance d. Since high quality data representation is required, the sensor devices SD A and SD B need to sample a high quality representation of the person, which will be sent to the rendering device of the other person. Therefore, high quality communication with reduced communication overhead is desired.

[0126] In the following, embodiments of high quality communication or interaction with low communication overhead achieved by compression are presented.

[0127] Figure 4 The block diagram schematically shows a network architecture for implementing some embodiments.

[0128] The transmitter (Tx) 22 is understood as a device (e.g., a 5G UE) for sensing or generating data to be compressed. The data or compressed data is transmitted to the network (NW) 20 via an access link (AL) (e.g., a 5G new radio (NR) radio link). In addition, the receiver (Rx) 28 (e.g., a 5G UE) is understood as a device for rendering data or compressed data. The data or compressed data is transmitted from the network 20 via an access link (AL) (e.g., a 5G NR radio link).

[0129] In addition, a network (NW) 20 is provided, which is understood to be any type of arrangement or entity for transmitting, storing, processing and / or otherwise processing data or compressing data. The network 20 may include multiple logical components distributed over multiple physical devices. In an embodiment, network edge servers 24, 29 and a core network (CN) 21 may be provided.

[0130] The network edge servers 24, 29 are understood to be devices that are physically close to a radio access network (not shown) and provide data processing and storage services for user equipment (e.g., UE) involved in the interaction or communication. The physical proximity provides very low latency communication with the user equipment. In an exemplary application, a sending edge server (TxES) 24 and a receiving edge server (RxES) 29 may be provided at the transmitter 22 and the receiver 28, respectively, and may be configured to provide storage capabilities for data and compression / decompression databases, compression / decompression functions, and negotiation functions for negotiation with peer devices.

[0131] Furthermore, the core network 21 is understood to include the remaining part of the network 20 for transmitting data between the transmitter 22 and the receiver 28 , possibly via respective edge servers 24 , 29 .

[0132] Additionally, shared storage (S) 23 may be provided as a virtual device representing memory shared by both transmitter 22 and receiver 28 and / or compression and decompression functionality of edge servers 24, 29. It may be physically located in one or more places, with multiple copies synchronized as needed.

[0133] Communication parameters are understood to be parameters that affect the performance of communication or place demands on communication. They may include at least one of latency, QoS, distance between communicating parties, computational requirements for processing communication, computational power for processing communication, memory requirements for processing communication, memory capacity for processing communication, available bit rate, number of communicating parties, etc. Some of these parameters are related to each other. For example, the latency between two user devices depends on the distance between the devices and also on other aspects, such as computational requirements and / or capabilities for processing communication. In particular, if the communication involves a predictive model, the communication latency may be affected by the available / required computing power of the two devices.

[0134] In some embodiments, without loss of generality, only some of the above communication parameters may be mentioned. For example, if the embodiment only mentions delay, this should be understood as delay or other communication parameters, especially other communication parameters that affect the delay of the communication link.

[0135] In an embodiment, sufficient image quality (e.g., realistic representation of a person) can be provided at the receiving end by using data compression. Such data compression can involve conventional data compression schemes. In addition, specific data compression schemes and corresponding devices or systems for compressing and decompressing data are described in the following embodiments. It should be noted that these embodiments, while beneficial to the specific applications mentioned herein, can also be implemented independently, for example, in a context other than virtual reality and for other applications.

[0136] Embodiments may relate to systems of the first type in which a compression device is intended to take some input data and store it efficiently in a storage unit. This may be useful in cloud-based settings where some data needs to be stored efficiently. Here, a compression device or encoder is used as a device (which may be a software product) that performs data compression. For example, a device may receive data and may be able to decompose (classify) the data into one or more data objects according to appropriate criteria, and then compress the data objects again according to appropriate criteria, storing the compressed data objects together with any necessary compression models and other metadata required for later reconstruction of the source data. Appropriate criteria may include considerations of the required (semantic) accuracy of the reconstruction, available storage space and processing resources for the reconstruction, and the like. For purposes of this disclosure, criteria may also include considerations of the semantic content of the data and data objects and the compression model to be used.

[0137] Optionally, a decompression device (or decoder) may be provided, which is understood to be a device (which may be a software product) that retrieves compressed data from storage and, using an appropriate (de)compression model, reconstructs and renders the original data with minimal semantic loss. For this purpose, a compression model library may be used, which is understood to be a database comprising tools and data objects for data compression, and which may advantageously be used for both compression devices and decompression devices. Subsets of the library's tools and / or data objects may be combined to form a compression model that is optimized in some way for a given sample or data type.

[0138] A storage unit is understood to be a device accessible to a compression device when storing and to a decompression device when retrieving, which stores compressed data and the accompanying compression model and metadata. It may also hold a compression model library. The storage unit may take many forms. For example, it may be a web-based server, or it may be a physical device such as a memory stick, a hard drive, or an optical disk.

[0139] Other embodiments may involve a second type of system, where a compression / transmission device is intended to exchange data with a second decompression / reception device in an efficient manner. This may be useful in a setting where a transmission device (e.g., a streaming service cloud) wants to efficiently share data with a reception device (e.g., a TV). In some cases, a transmission device (e.g., Figure 4 The transmitter 22 in the example may have a sensor capable of sensing / capturing data, such as VR glasses or a mobile terminal or user equipment. In some cases, the receiving device (e.g., Figure 4 The receiver 28 of the edge server may have a sensor capable of reproducing / rendering the data, such as VR glasses or a mobile terminal or user device. In some cases, the edge server (e.g., Figure 4 The edge servers 24, 29 in the example may be associated with one of the send / receive devices that take over and / or share some of the functions. In some cases, some of the send / receive devices may be part of a telecommunications system such as a 5G system.

[0140] In conjunction with other embodiments of the second type of system, a compression sending device is understood as a compression device that compresses data in a substantially streaming manner, typically taking into account latency or computational overhead or communication overhead at the sending / receiving device as part of its compression criteria, and delivers the compressed data to a transmission channel. In addition, a decompression receiving device is understood as a decompression device that decompresses the data when the data arrives at the transmission channel, typically rendering the data in real time, and typically taking into account latency or computational overhead or communication overhead as part of its rendering criteria.

[0141] An (optional) edge server may be provided next to the compression sending device and may be able to assist the compression sending device by compression (post) processing, immediate provision or updating of compression models, or otherwise ensuring timely delivery of compressed data. Similarly, an (optional) edge server may be provided next to the decompression receiving device and may be able to assist the decompression receiving device by decompression (pre) processing, immediate provision or updating of compression models, or otherwise ensuring timely rendering of decompressed data.

[0142] In addition, (optional) sensors (e.g., audio, video) may be provided at the compressed sending device and may be configured as a device or array of devices for capturing some aspects of the scene, typically but not necessarily in real time. Examples include cameras, microphones, motion sensors, etc. Some devices can capture stimuli outside the range of human perception (e.g., infrared cameras, ultrasonic microphones) and can "down-convert" them into human perception form. Some devices may include an array of sensor elements that provide an extended or more detailed impression of the environment (e.g., multiple cameras capturing a 360° viewpoint, multiple microphones capturing a stereo or surround sound sound field). Sensors with different modalities may be used together (e.g., sound and video). In this case, different data streams need to be synchronized. The compressed sending device equipped with sensors may be VR / AR glasses or a simple UE.

[0143] In addition, (optional) rendering devices (e.g., audio, video) can be provided at the decompression receiving device, and can be devices or arrays of devices that render some aspects of the scene, usually in real time. Examples include video displays or projectors, headphones, speakers, tactile transducers, etc. Some rendering devices may include arrays of rendering elements that provide an extended or more detailed impression of the captured scene (e.g., multiple video monitors, speaker arrays for rendering stereo or surround sound audio). Rendering devices with different modalities (e.g., sound and video) can be used together. In these cases, the presentation subsystem must ensure that all stimulus channels are presented synchronously.

[0144] Furthermore, an (optional) communication manager may be provided in the second type of system, which may be a centralized or distributed entity that manages communications. The goal of the communication manager may be to optimize communications in terms of latency, overhead, etc. The communication manager may be an entity in the communication network such as 5GS, or may be an external entity such as an application function.

[0145] As another (optional) element of the second type of system, the compression model library may contain information, such as data, machine learning (ML) models used to derive the data, etc., for reconstructing the data based on, for example, hints.

[0146] In the following embodiments (at least some of which may be combined to further improve performance), compression and reconstruction of data may be based on hints to a text-to-image model (such as a latent diffusion model), which may be learned on the fly to represent previously unseen objects without the need to retrain the entire reconstruction model. This technique ("textual inversion") may be done quickly and iteratively, as described by Rinon Gal et al. in "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion" (available at https: / / textual-inversion.github.io / retrieved from the address described in.

[0147] Furthermore, in an embodiment, the compression and reconstruction of the data may be based on a system in which generative compression based on text inversion is guided by an input image so that the reconstruction remains similar to that image, as described by Zhihong Pan et al. in “EXTREME GENERATIVE IMAGE COMPRESSION BY LEARNING TEXT EMBEDDING FROMDIFFUSION MODELS” (available at https: / / arxiv.org / pdf / 2211.07793.pdf as described in the retrieval).

[0148] Additionally, in embodiments, image classification can be run quickly on low-power devices, as described by Salma AbdelMagid et al. in “Image Classification on IoT Edge Devices: Profiling and Modeling” (available at https: / / arxiv.org / pdf / 1902.11119.pdf as described in the retrieval).

[0149] Typically, diffusion models are good at reproducing items that form part of their training dataset, but poor at reproducing items that do not form part of their training dataset. Since training datasets can be obtained from the public internet and diffusion models are expensive to retrain, this leads to problems reproducing a priori unknown inputs.

[0150] For example, a diffusion model may be able to recreate images of people in general based on a cue such as “young person”, but may not be able to recreate images of any one specific person (except in some edge cases such as celebrities).

[0151] This loss of realism between the regenerated output and the original observation is called the "semantic loss", and it differs from the distortion introduced by traditional codecs in several ways; in particular, it is more dependent on the original object being observed.

[0152] In addition to semantic loss, spatial and temporal stability can be an issue. In a related art (Neural Radiance Fields or NERF), recent work has addressed techniques for improving spatial and temporal stability.

[0153] Recent techniques (e.g., as described above) have attempted to address the problem of semantic loss. Prior art in this area is represented by text inversion, i.e., dynamically learning embeddings to represent previously unseen objects without having to retrain the entire diffusion model. Thus, guidance images are used to ensure that the learned embeddings adequately represent the observed reality (as represented by the guidance images).

[0154] Figure 4 The basic idea of ​​the present invention is described to achieve low-latency interaction between devices or people in remote locations while reducing communication overhead, such as in virtual reality applications.

[0155] According to a general definition of the invention, a first person (or device) may interact with a second person (or device), for example in a virtual reality application, through a predictive model of the second person, where the predictive model is located near the first person. The predictive model may then predict the state or sensor input at time t based on the sensor input sampled at the previous time, for example, time t(0) = td*c, t(1) = td*cT, ..., t(N-1) = td*c-(N-1)*T, where c is the speed of light and T is a given sampling period. Different types of predictive models may be applicable.

[0156] This method allows:

[0157] First, low latency, because instead of direct interaction, person A (B) interacts with a predictive model of person B (A). The predictive model of person B (A) predicts the actions of person B (A) based on the input collected by sensor device B. (The output of) the predictive model of person B (A) is used in the rendering device of person A (B).

[0158] The model of person A is built when the person joins the virtual reality. The model can be deployed to a suitable location when person A wants to interact with another person B. The suitable location is preferably close to person B (e.g., running on a local edge server).

[0159] Second, data compression can be achieved by: a) downloading a complex model of the person at a suitable location during initialization; b) limiting the sensor data that needs to be sampled for a suitable prediction model that leads to a realistic representation of the person, for example, as in a generative model.

[0160] Another idea underlying the present invention relates to synchronization of (predicted) communication flows from different user equipment at different locations by:

[0161] First, the (communication) parameters (eg, delay) between users are determined.

[0162] Second, the streams are synchronized based on (communication) parameters.

[0163] Third, optionally, this synchronization is applied to the use of the predictive model.

[0164] Communication parameters are parameters that affect communication performance or impose requirements on communication. Communication parameters may include:

[0165] - Delay,

[0166] -QoS,

[0167] - the computational requirements for handling communications,

[0168] - computing power to handle communications,

[0169] - memory requirements for handling communications,

[0170] - memory capacity for handling communications,

[0171] - available bitrate,

[0172] - the number of communicating parties,

[0173] -…

[0174] Note that some of these parameters are related to each other. For example, the latency between two devices depends on the distance between the devices, but also on other aspects, such as the computing requirements and / or capabilities for processing the communication, or the computing power for performing the rendering of the received data. In particular, if the communication involves a predictive model, the communication latency may be affected by the available / required computing power of the two devices.

[0175] Note that in some embodiments, only some of the above communication parameters may be mentioned without loss of generality. For example, if an embodiment only mentions latency, this should be understood as latency or other communication parameters, especially other communication parameters that affect the latency of a communication link.

[0176] Note that in some embodiments, beside the communication parameters, other parameters may also be considered in the adaptation of the communication functionality, such as transmission time, required data rate, ML model, compression ratio. In particular, the parameters may be related to the (relative) pose, speed or orientation. These parameters are relevant because if an object / person (e.g. involved in a virtual reality session) moves closer / further away at a higher speed, a higher communication rate may be required to perform an accurate rendering. Similarly, a more powerful model may be required to perform better predictions. Similarly, if the relative orientation of a person / device (e.g. involved in a virtual reality session) changes, a higher communication rate may also be required to perform a better rendering or a prediction of the data required for rendering.

[0177] General description of the system

[0178] The present invention is described in the context of a system focused on implementing applications such as: virtual reality, or an immersive teleconferencing system, or immersive real-time communication between several persons A, B, ..., I, ..., N.

[0179] Each person is surrounded by devices D connected to each other via communication links (e.g., 5G communication links), such as Figure 5 As shown. The communication link from device i (person i) to device j (person j) is characterized by a set of (communication) parameters P_ji, which may include parameters related to the physical distribution of the devices, such as distance, speed, orientation or actual communication parameters such as average latency (depending on distance), jitter, bandwidth, reliability, QoS requirements, privacy requirements, etc. Note that some of these parameters may be affected by, for example, allocating more resources (e.g., reliability), but other parameters (e.g., latency) cannot be modified. The set of communication parameters can be represented as a rectangular matrix.

[0180]

[0181] The entire system uses the following definitions related to IEEE 1918.1 and Figure 1. The user in the region of interest (ROI) is surrounded by a set of tactile devices (TD) linked to the tactile edge (TE). The TD may include rendering actuators and / or sensors. The TD rendering actuator may have the task of creating a virtual reality environment around the user, and may be VR glasses, 3D TV, holographic devices, etc. The sensor TD is a device responsible for capturing the user's movements and / or environment, and may include a camera, an audio device (such as a microphone), a tactile sensor, etc. The TD in the ROI can be connected to the user's TE, for example, by wire or wirelessly. In the wireless case, the UE is connected to a base station (such as a 5G gNB) or a WiFi access point. The networking infrastructure and computing resources of the TE can be co-located in the ROI or located in a near-edge server (less than dmaxedge) to ensure a fast response.

[0182] Typically, TD can be the UE in the 5G system.

[0183] In order to help implement the embodiment of the present invention, a communication function may be introduced:

[0184] First, delay-based flow synchronization (LBFS): It is a function that can be run in the equipment in TE. It can also be deployed in the receiving TD, which can determine the communication parameters with the sending TD (or TS) and synchronize the communication flow based on those communication parameters (especially the relative delay between TD (or TE)).

[0185] Second, a configurable prediction model (LDCPM) of the environment / human latency-related / communication parameter-related in the virtual reality session is run in different TEs (e.g., provided by an edge application at the TE).

[0186] Third, the model management and configuration functionality enables one or more of the following: (1) registering a common model of ROI / device / person in TE, (2) storing it in a database, and (3) deploying (reconfigured) LDCPM when determining communication parameters.

[0187] This function is Figure 6 , where:

[0188] First, LDCPM management is shown as part of the 5GC system, although it can also be a fully or partially external function. LDCPM models can be shared by third-party applications, and LDCPM management can store generic LDCPM models in applications and deploy (configure) LDCPM to TD / TE when needed. For example, in some embodiments, LDCPM can run on TE and LBFS can run on TE. For example, in some embodiments, delay-related / communication parameter-related configurable prediction models can be determined centrally, while in other embodiments, delay-related / communication parameter-related configurable prediction models can be determined at the entity that executes them (e.g., the receiving TE or TD) based on the currently measured communication parameters.

[0189] LBFS or other related building blocks shown in the present invention (e.g., Figure 7 The building blocks that can configure / determine the delay / distance / parameters between TEs) can be as follows Figure 8 The blocks shown are summarized as being able to configure / determine inter-TE (or inter-TD) (communication) parameters (eg, orientation of the device) and perform actions based thereon. For example, synchronizing communication flows as in some embodiments of the present invention.

[0190] The sending and receiving TE may be identical, ie the sending and receiving TD may be in the same TE, and therefore in close proximity.

[0191] User creation, TD registration.

[0192] The application may provide a user interface for the user / ROI to register in the TSM, for example.

[0193] Registration can be done through the app itself, or through TSM.

[0194] Registration may include registering the TDs that a user / ROI has.

[0195] Registration may involve registering the TD as part of the communication infrastructure, e.g., registering as part of the 5G System (5GS) to access functionality provided by the 5GS, such as quality of service, low latency, edge computing, or synchronization of communication flows.

[0196] During registration, the TSM can assign a TE to a user / ROI / TD that is close to or suitable for its communication parameters.

[0197] In some cases, a sensor TD may generate output to be fed to a TD for rendering in the same ROI.

[0198] TD registration may also be done during UE registration in 5GS, for example, as part of initial primary authentication. The TD may expose its capabilities or NF in 5GS, for example, the capabilities of the TD may be looked up which may be stored in, for example, UDM or UDR or other AF. Based on the location of the TD, which may be determined by the 5GS / TSM, the TD may be assigned to a given TE. Based on the capabilities of the TD, the 5GS / TSM may assign computing or communication capabilities to the TD.

[0199] Model registration, creation, deployment and use

[0200] The model of a user or ROI may refer to one or more generic models of a ROI / device / person in TE, for example, a model in terms of the TD sensors involved, or a model in terms of the (person) features that need to be rendered, etc.

[0201] The model is able to generate a suitable (predictive) representation of the ROI or person and may rely on different types of artificial intelligence / machine learning models / networks, such as generative models.

[0202] User / ROI registration may involve the creation and registration of: a (generic) prediction model M of the user's ROI, or the user himself that is able to predict the state of the ROI (or user). For example, if the sampling periods are uniformly distributed, then given N sensory inputs of the ROI / person in the past (e.g., samples generated by sensor TD at times (t-cd, t-cd-T, t-cd-2T, . . . , t-cd-NT), the state of the ROI / user at time t can be predicted.

[0203] Other sampling distributions are possible, for example, samples may be more dense at recent moments and less dense at earlier moments:

[0204] (t-cd,t-cd-T1,t-cd-2T1,…,t-cd-NT1 / 2,t-cd-NT1 / 2-T2,t-cd-NT1 / 2-2T2,…,t-cd-N(T1+T2) / 2

[0205] Among them, T1 <T2。

[0206] Typically, a model can be a function

[0207] M(D,cd,T,N)

[0208] Where D represents data sampled at the sensor TD (in the transmitting TE) and used in the model of the transmitting user / ROI / TD / TE deployed in the receiving TE, cp also refers to the communication parameters between the transmitting TE and the receiving TE (e.g., cd, i.e., latency L), T represents the sampling frequency (or multiple sampling frequencies) of the sensed data, and N represents the number of samples used for inference in the model.

[0209] The model may also be based on an AI / ML model, such as a generative model that allows for the derivation / generation of a given data output (e.g., audio / video / …) based on a (textual) prompt. The prompt may represent a user and may be used to generate a data representation of the user that may be rendered. The prompt may include metadata indicating the position / orientation of the data representation of the user (to be generated). The model definition also fits the previously mentioned functions, where D in this case refers to the prompt+metadata information. For example, a prompt ["Bob", "motion_vector", "orientation", t0] may be a sensor sample and indicate that the generative model should be used to obtain a data representation of Bob, who is moving with a given "motion vector" and has a given "orientation" at time t0. For example, a given model may utilize N such data samples.

[0210] Models can be created by at least one of the following:

[0211] First, place the sensor (TD) on or around the user.

[0212] Second, the user is asked to perform certain movements while

[0213] Third, measuring the actual movement, for example using additional (calibrated) sensors,

[0214] Fourth, use TD to measure movement,

[0215] Fifth, the calibrated measurements are used to train a model that uses the output of TD as input.

[0216] The prediction model may refer to a deep learning model, such as a recursive neural network model, such as a long short-term memory (LSTM) model. The prediction model may refer to a single model or multiple models for the entire TE, for example, a model according to TD in TE or a model according to users (features) in TE.

[0217] In some cases, the trained model may be trained to make predictions "far" into the future, i.e., it represents a large delay L = c.dmax linked to the maximum distance dmax between two TEs. For such a model, the TD in the sending TE sends a sample of the environment to the TD in the receiving TE. The receiving TE will already be configured with the prediction model of the sending TE and will use N received samples, e.g. the last N received samples, when feeding the model. If the sending and receiving TEs are at a distance d < dmax, the receiving TE will have to defer the stream generated by the model c(dmax-d).

[0218] In some cases, the trained model can be trained to make predictions at any time in the future, i.e., it represents an arbitrary delay L linked to the distance d between two TEs or the computational load of those TEs. For such a model, the TD in the sending TE will send samples of the environment to the TD in the receiving TE. The receiving TE will already be configured with the prediction model of the sending TE and will use up to N received samples, e.g., the last N received samples, when feeding the model. In this case, the generated data / stream can be used directly for the receiving TE.

[0219] In some cases, the number of samples required in the prediction (input to the model) may be fixed, however, if the delay L (i.e., d) is small, the number of samples exchanged may be lower, for example, only every second sample sampled at the transmitting TD may be transmitted. The receiving TE may infer the missing samples (e.g., by interpolation) before feeding them into the model. This approach may allow for a reduction in communication overhead. Figure 7 A possible embodiment of the method is depicted.

[0220] exist Figure 7 In the example, the TD sensor samples the environment in the transmitting TE with a period T. Given a delay (or distance, or communication parameter, or other parameter of the sampling scenario) value configured (e.g., by the TSM or by the application) or determined (e.g., by the transmitting and receiving TEs in a distributed manner) between the transmitting / receiving TEs, the transmitting TD can determine a compression parameter, such as a subsampling frequency for samples sampled by the transmitting TD sensor. The transmitting TD then sends the compressed samples to the receiving TD. The samples arrive with a delay of dc seconds. The receiving TD has been configured or has determined the same delay (or distance) between the TEs, and can use this delay to decompress the input signal, for example, to determine and apply an interpolation rate for the received signal. The model then uses N samples corresponding to a time period of N*T seconds to infer a signal that can control the actuator at the receiving TD at the current time (i.e., without delay).

[0221] For example, if the TD sensor returns a prompt for the generative model such that the prompt can be used to regenerate the representation of the remote user, then if the latency is high, the sending TD (TE) can choose to send samples more frequently so that the receiving party can perform a better prediction of the future state of the remote user.

[0222] For example, if the TD sensor returns a prompt for the generative model such that the prompt can be used to regenerate the representation of the remote user, then if the user is moving fast or if the user is quickly changing his orientation (e.g., rotating), the sending TD (TE) can choose to send samples more frequently / at a higher data rate so that the receiving party can perform a better prediction and / or better rendering of the future state of the remote user. In some cases, the number of samples required in the prediction may be variable. For example, if L (i.e., d) is small, a smaller number of samples N can be used in the deployed model. This can be achieved if the sending TE has multiple models trained for different (communication) parameters (e.g., different latency L / distance d) and it selects the most suitable model based on the communication parameter (e.g., distance). This can reduce the CPU requirements at the receiving TE. An alternative is for the sending TE to train a general model which is then converted into an adapted / compressed / customized model when deployed at the receiving TE characterized by certain communication parameters (e.g., distance d (latency L)). In a potential instantiation, the general model can take N samples as input, but when d < dmax, the compressed / customized model can take < N samples as input. This can be achieved, for example, by combining a decompression algorithm, for example, with Figure 7 the general model into a single block "adapted model" as shown in Figure 8 For example, once the TE (or TD) - to - TE communication parameter (e.g., latency / distance) is determined, the adapted / compressed / customized model can be created at the sending TE (or sending TD) and directly deployed to each receiving TE (or receiving TD). Once the TE (or TD) - to - TE communication parameter (e.g., latency / distance) is determined, the compressed / customized model can be created at a central entity and the central entity can trigger the deployment to each receiving TE (or receiving TD).

[0223] The sending and receiving TE can be the same, i.e., the sending and receiving TD can be in the same TE and are thus close. In Figure 7 the building block capable of configuring / determining the TE - to - TE latency / distance as shown in Figure 8 can be generalized to a block capable of configuring / determining the communication parameter (e.g., orientation of the device) between TEs (or between TDs or within a TE).

[0224] There can be multiple embodiments to obtain such an adapted / compressed model, for example:

[0225] First, the interpolation block is connected to Figure 7 A combination of common models in , or

[0226] Second, use some technique as shown in Figure 9, or

[0227] Third, make the model predict the output T seconds later and apply it multiple times when the delay is k.

[0228] In some cases, the model may have an input of N and is designed to make a prediction given N samples at a given time cd, such as Figure 9a If the transmitting and receiving TEs are further away from cd, the receiving TE may choose to mask several (e.g., k) inputs of the model (which indicates that they are unavailable) and use a simplified model with Nk input samples and give a prediction of cd+kN seconds. This Figure 9b where k = 3. Obviously, in this case, the accuracy will decrease because some inputs are lost. If the transmitting and receiving TEs are very close, the receiving TE can choose to mask several (e.g., m) inputs of the model (which indicates that they are unavailable) and use a simplified model with Nm input samples. This is Fig.9c and Figure 9d , where m=11 and m=9. Figure 9a The model in can be a generic (highly accurate) model that can be reused for multiple transmitting TEs at multiple receiving TEs at different distances by masking or padding the input.

[0229] exist Figure 9a -d, the masked input is shown as having an input value of 0, however, other values ​​may be possible, e.g., the same value as the previous unmasked input, an interpolation of the unmasked input, etc.

[0230] This technique can also be used for data compression, for example, if the receiving TE is close, the transmitting TE can decide to only send every second sample from the TD sensor and inform the receiving TE that it must mask every second input in the model.

[0231] This technique can also be used to reduce a generic model to a compressed model when some of the inputs are masked, since operations can be combined taking into account the fact that some of the inputs are (always) masked.

[0232] In various embodiments, the general model may be, for example, an LSTM network capable of predicting the value for a given fixed delay T, for example, as may be represented in Fig. 10. In this embodiment:

[0233] If the receive TE (or TD) is d = T / c away from the transmit TE (or TD), a single iteration is required.

[0234] If the receive TE (or TD) is d = 2T / c away from the transmit TE (or TD), two iterations of the model are required, with the output of the first iteration used as input in the second iteration.

[0235] If the receive TE (or TD) is d = 3T / c away from the transmit TE (or TD), three iterations of the model are required, with the output of the first iteration used as input in the second iteration, and the outputs of the first two iterations used as input in the third iteration.

[0236] In general, if the receive TE (or TD) is d = kT / c away from the transmit TE (or TD), k iterations of the model are required, where the output of the first iteration is used as input in the second iteration, ..., and the output of the first (k-1) iterations is used as input in the final iteration.

[0237] Additionally or alternatively, in other prediction models, hints may be used to regenerate content on the receiver side. Hints may have been extracted on the sender side based on the input content. If the prediction model is shared, the same content may be generated on the receiver side based on the hints. If the content is an object or a person, and the object or person moves at a given speed, for example, the prediction model on the receiver side may take into account the speed of the person and the latency of the communication to predict where the object / person / etc. will be located.

[0238] For example, suppose the transmitter observes Oscar, a person on the left side of the image, and Oscar moves to the right at a speed of 10m / s. Suppose the transmitter can extract OscarPrompt from the image. Suppose the image has a width of 3m and a pixel width of 1920 pixels. Suppose the receiver receives data from the transmitter with a delay of 30ms. Suppose the transmitter sends "OscarPrompt, Current Position: left_size; Current_Speed: 10m / s" to the receiver, the receiver uses the prompt "Oscar" to reproduce the image of Oscar, and takes into account the speed information and delay information (from the transmitter to the receiver) to show / display / render the image of Oscar starting with pixels (10m / s*0,03s)*1920 pixels / 3m=192 pixels.

[0239] In this example, if the receiver receives data from two transmitters, such as:

[0240] "OscarPrompt", current position: left_size; current_speed: 10m / s", delay 30ms

[0241] "AlicePrompt", current position: left_size; current_speed: 10m / s", delay 10ms

[0242] Even if the data arriving from Alice is received 20ms earlier than the data arriving from Oscar, the receiver will show both Oscar and Alice's images on top of each other. The reason is that the model takes into account the delay / latency of the communication when synchronizing the communication streams (e.g., predicting the current position of the person / object / content to be rendered / displayed).

[0243] Communications infrastructure configuration

[0244] In an embodiment, a metaverse application utilizing a communication infrastructure may be able to configure the communication infrastructure.

[0245] This configuration can be done through TSM which is used to coordinate the underlying network.

[0246] 5G (or 6G) TSM may exist in 5GS (or 6GS).

[0247] When both entities are present, the TSM will interact with the 5G TSM. This means that the TSM can act as the primary entity for coordinating the communication of applications (e.g., virtual reality applications when the 5GS TSM is responsible for the 5GS). This is also relevant in cases where multiple 5GSs are involved (e.g., a 5GS of the home network and a 5GS of the visited network). It can also be that the 5G TSM of the home network acts as the primary entity and the 5GS TSM of the visited network "reports" to it (the 5GS of the home network).

[0248] This configuration can be done through policies.

[0249] The policy may include configuration items for each TD in each TE (eg, that may be involved in a communication link or session).

[0250] The application may add entries to the policy corresponding to a new TD or TE, for example, each time / when a new TE (TD) joins a (new) (virtual reality) communication session, authorization changes, etc.

[0251] The TSM may distribute the policy of the new TD(TE) to all existing TD(TE) already involved in the (meta-reality) communication session.

[0252] The TSM may distribute a policy to the new TD(TE) including entries of all existing TD(TE)s already involved in the (meta)communication session.

[0253] The configuration may be a one-time configuration, or may be a virtual reality session configuration for a virtual reality session between several TEs (eg, several users (A, B, ..., i, ...)).

[0254] Configuration may include policies that specify, for example, at least one of the following:

[0255] First, the QoS target depends on, for example, the number of users, relative latency,

[0256] Second, the resynchronization delay applicable to the data stream originating from the transmitting TE (or at the transmitting TD in the transmitting TE) and the resynchronization stream applicable to the receiving TE (or the receiving TD in the receiving TE): TAU_i, TAU_j, TAU_bm, TAU_am, as explained in other embodiments.

[0257] Third, there is a need to continuously monitor the delay between TEs and the update rate of the parameters, as explained in other embodiments.

[0258] Fourth, the delay requirement for each TD in the TD in the TE, ..., enables the compression or model to be adapted accordingly.

[0259] Fifth, the prediction model of each TD in TE, …, enables TSM to deploy its model or compressed model to other TD / TE in the communication session.

[0260] Sixth, the need for QoS balancing, and if applicable, methods as in other embodiments.

[0261] Similarly, the communication infrastructure may also inform the virtual reality application of communication parameters or other parameters (such as the relative position of the TD) and / or configure the virtual reality application.

[0262] The user at TDi / TE / i starts the application with learning / configuration of parameters (e.g., communication parameters)

[0263] In an embodiment, when a user starts a virtual reality application involving one or more (remote) users, the communication infrastructure needs to learn a set of parameters, such as communication parameters, with the rest of the users / ROIs. This can be done in two main ways:

[0264] First, end-to-end learning from a new TD (or TE) to other TDs (or TEs): In this case, the new TD of the TE is configured by the TSM or application with the identities of other already existing TDs / TEs involved in the current communication / virtual reality session. The existing TDs / TEs are also notified of potential new TDs / TEs. The existing TDs / TEs can also retrieve information about the new TDs / TEs, for example, from a library. The TDs (or TEs) can perform distributed learning of parameters (e.g., communication parameters (e.g., distance or delay)), i.e., TD to TD or TE to TE. For example, two remote TDs can measure the round-trip time and divide it by two to determine the delay (distance) of the communication link. For example, two remote TDs can share their current speed / orientation with each other so that they can derive, for example, their relative speed / orientation.

[0265] Second, TSM supported based on communication of parameters (e.g., latency from TD / TE to (5G)TSM): In this case, a new TD of a new TE is registered in the system, and parameters (e.g., latency) from / to the (central) networking infrastructure (e.g., the relevant points through which (all) communications of the (all) TEs involved pass) are measured from the new TD / TE. For example, the remote TD can measure the round-trip time with the central networking infrastructure and divide it by two to determine the latency (distance) of the communication link; alternatively, the central networking infrastructure can perform this action. Given this measurement, the TSM can calculate the end-to-end communication parameters between each pair of TD / TE, and: a) configure the new TD of the new TE or the new TE with the communication parameters of an already existing TD / TE, and b) configure the already existing TD / TE with the communication parameters of the new TD / TE. For example, the TSM can collect information about the orientation of the TDs and determine their relative orientations. The TSM can then use this information to adapt the communication parameters.

[0266] Other parameters and / or communication parameters may also be learned in a similar manner and / or may be exposed, for example, the (available) computing power of the TE / TD may be shared with the TSM.

[0267] The TSM may use the learned parameters and / or communication parameters, or expose / exchange / share the learned parameters and / or communication parameters with an application (e.g., an application function within the 5G system or outside the 5G system).

[0268] The application started by the user at TDi / TE / i triggers the deployment of the prediction model at the remaining TD / TE

[0269] In an embodiment, when a user starts a metaverse session, a model of the user's TE / TD may be shared by the application with the TSM.

[0270] The TSM can then determine which TDs / TEs already associated with the metaverse session require a model.

[0271] The TSM can directly configure the TD / TE with the parameters between them (eg, communication parameters / delay / distance), or let them determine the parameters in a pairwise manner.

[0272] TSM can then choose appropriate compression parameters.

[0273] TSM can compute a compressed version of the model to deploy based on, for example:

[0274] First, communication parameters / delay / distance between TD / TEs: For example, if the communication parameters between the source TD / TE and the target TD / TE indicate a large distance / delay, the deployed model may be more complex in terms of the number of past samples used in the prediction.

[0275] Second, computing power: For example, if computing power is insufficient for a highly accurate model, a simplified model is deployed that requires fewer computing resources for prediction.

[0276] Third, other parameters, such as relative pose / velocity / orientation between TD / TE.

[0277] When the model has been successfully deployed in all target TD / TEs, the user / networking infrastructure may be notified as this may serve as an indication that communication may begin.

[0278] Once this happens, the virtual reality application is also notified so that the user can start a virtual reality session.

[0279] Alternatively, the TSM may expose the parameters (eg, communication parameters or a subset thereof) to the metaverse application / user which will then derive an appropriate model.

[0280] Operation - Unicast vs. Multicast Streaming

[0281] Unicast communication flows require each sensing TD (N devices) to maintain NM unicast flows to each actuator / rendering TD (M devices). This may become less efficient as N and M increase.

[0282] A more efficient approach includes a multicast approach, where each sensing TD multicasts its stream, and the stream is distributed to each of the subscribed rendering TDs. This involves N multicast streams, although it is still important to consider that the multicast streams may arrive at different rendering TD / TEs at different times, and those TD / TEs that receive the multicast stream earlier may use, for example, the compression model of the sending TD / TE, while those TD / TEs that receive the multicast stream later may need, for example, a less compressed model.

[0283] Thus, a multicast stream may include multiple data streams tailored for compression models with different compression ratios.

[0284] Synchronization of operation-data flow

[0285] In one embodiment, the sending TE synchronizes outgoing communication flows originating from the sensing TD in the TE. This may involve, for example, the TD or TE being configured with a strategy for determining how to synchronize communication flows originating from multiple sensing TDs in the TE, which involve sensors with different sampling frequencies, such as video, audio, or tactile sensors. These sensors may not be completely synchronized, or the sampling moments may not be aligned, and therefore, the networking infrastructure in the TE may resynchronize these communication flows between them. To this end,

[0286] First, each of the data flows i originating from TD i in TE may be delayed by time TAU_i;

[0287] Second, the TAU_i value can be configured through policies;

[0288] Third, TE / TD can be configured with sampling instants;

[0289] Fourth, the TE / TD may be configured with communication resources to allow synchronous retrieval of sampled data;

[0290] Furthermore, TE j may be responsible for aligning all its outgoing flows with the rest of the data flows in the system by delaying all data flows by a delay TAU_j.

[0291] To achieve this, the virtual reality application executed at the sensing TD may need to expose the data / parameters (e.g., related to sampling frequency and / or time (sampling time) and / or clock, etc.) of each sensing TD in the sensing TD to the underlying communication infrastructure, such as 5GS. This can be exposed through the TD itself, through third-party application communication with TSM, etc.

[0292] Once this information is available, a policy can be deployed to the networking infrastructure (e.g., 5G) that determines how to synchronize outgoing communication flows and, for each of the outgoing communication flows, the delay values ​​TAU_i and TAU_j to apply at the sending TD / TE.

[0293] In another embodiment, the receiving TDs in the receiving TE are instructed to synchronize the incoming streams. Stream synchronization may also be applied in the TE for all receiving TDs.

[0294] This may imply that the TD (or TE) may be configured with a policy for requiring the delayed TAU_bm and / or TAU_am to be applied to each incoming flow.

[0295] The delay TAU_bm may be applied before passing the data to the (compression) model for inferring the (control) signal for a given TD.

[0296] This delay may allow for fine synchronization of the input from the received input data stream.

[0297] The delay TAU_am may be applied to the (control) signal obtained from the model in the following cases for example:

[0298] (1) applying the general model for generating a (control) signal corresponding to the maximum delay / distance or the current time at the receiving TE, while the receiving TE is located at a distance d less than dmax from the transmitting TE, and therefore TAU_am = c(dmax-d), or

[0299] (2) The predicted signal is slightly in the future.

[0300] As in the previous embodiments, all communication flows originating from the sending TE may also need to apply the delay TAU_j (if this delay is applied in the receiving TE, TAU_j may not be required in the sending TE).

[0301] The input of the model can depend on the communication parameters between the receiving TD(TE) and the sending (remote) TD / TE, such that the output of the model (of the remote environment) corresponds to the current state of the remote environment at the sending TE.

[0302] All or some of the configuration parameters, such as the delay, can be configured in the TD or TE.

[0303] All or some of the configuration parameters (e.g., the delay) can be stored, for example, in a database, in the communication infrastructure, or in an external virtual reality application.

[0304] All or some of the configuration parameters (e.g., the delay) can be exchanged between the communication infrastructure (e.g., a telecommunications network) and the external virtual reality function or exposed to the external virtual reality function.

[0305] Operation-Stream synchronization based on relative delay without prediction model

[0306] In related embodiments, based on a configuration policy, in particular based on communication parameters, in particular based on the relative distance (or time delay), the incoming flows (originating from multiple TEs) are synchronized.

[0307] Specifically, if TD1 at (TE1) receives two communication flows F21 and F31 from TD2 at (TE2) and TD3 at (TE3) characterized by time delays L21 and L31 such that L21 < L31, then the flow F21 is delayed by a period of time (L31 - L21) so that it is synchronized with the flow F31.

[0308] Operation - Relative latency based stream synchronization with prediction model

[0309] In related embodiments, the actions of user 1 in TE1 will need to be rendered with a delay L31, since the time delay within TE1 can be considered 0. This may cause interference to the user.

[0310] Therefore, instead of delaying the communication flow that arrives earlier at the TE (e.g., the communication flow F21), the prediction model linked to the deployed TE2 in TE1 is responsible for predicting the timing of the state of TE2 at the current time given the previous samples received at TE1, as indicated in other embodiments (e.g., embodiments related to model registration, creation, deployment, and use).

[0311] Operational - Continuous delay monitoring / prediction (e.g., for communications via satellite links)

[0312] In some cases, the latency / distance between users / ROIs is not static, but varies due to the underlying networking infrastructure. For example, the communication link may be based on a satellite link, and if the satellite is not in geostationary orbit, the distance between the satellite links may be variable. Similarly, the communication link may suffer from congestion, resulting in higher latency. Therefore, there is a need to handle such variable latency communication links.

[0313] To address this need, in an embodiment that may be combined with other embodiments, the following methods may (alone or in combination) be required to be (periodically) applied, for example, at a frequency specified in a policy configured by the TSM:

[0314] First, the UE or networking infrastructure or 5GS keeps monitoring the communication parameters (e.g., end-to-end delay) and updates the communication strategy in a distributed manner. For example, two remote TDs can measure the round-trip time and divide it by two to determine the delay (distance) of the communication link.

[0315] Second, the communication infrastructure (e.g., 5GS) maintains monitoring and / or predicting communication parameters (e.g., end-to-end latency) between UE / networking infrastructure, and / or makes communication parameters (e.g., end-to-end latency) between UE / networking infrastructure available to the UE or the networking infrastructure.

[0316] Third, the communication infrastructure (e.g., 5GS) may need to store expected / average communication parameters (e.g., latency, congestion level, jitter level, ...) of a communication link (e.g., between two routers) in order to estimate the communication parameters of the end-to-end communication link based on the selected communication path.

[0317] Fourth, the communication infrastructure (e.g., 5GS) can expose expected or predicted end-to-end communication parameters to application functions (e.g., virtual reality application functions) located in the communication infrastructure (e.g., 5G CN) or outside the communication infrastructure. The application function can use the communication parameters to calculate a customized model that can be shared and deployed in the communication infrastructure.

[0318] Fifth, the communication infrastructure (e.g., 5GS) may expose expected or predicted end-to-end communication parameters (e.g., latency) with / between the remote parties of communication to the entity (e.g., TE) running the prediction model, so that the entity may use the end-to-end communication parameters to:

[0319] Adapting a base model of the remote party to a customized model of the remote party, the base model being usable for computing prediction values ​​based on an input stream received from the remote party, and / or

[0320] The predicted value is calculated using the base model of the remote party in conjunction with the input stream received from the remote party.

[0321] Sixth, the communication infrastructure (e.g., 5GS) can coordinate a base model of a first communication party, wherein use of the base model is (continuously) adapted to communication parameters between the first communication party and a second communication party served by the communication infrastructure, e.g.

[0322] The communication infrastructure may adapt or remain adapted to the base model to be customized based on real-time communication parameters between the first communication party and the second communication party.

[0323] The communication infrastructure may adapt or maintain the input adapted to a base model generated from an input data stream received from the second communication party based on real-time communication parameters between the first communication party and the second communication party.

[0324] The above techniques can also be applied to / used for other parameters, such as the relative pose / velocity / orientation of the TD.

[0325] Operation - continuously monitors transmitter / receiver relative position and adapts communication settings

[0326] In some scenarios, if the relative posture of the transmitter UE (such as TD) / receiver UE (such as TD) changes quickly, a higher data rate is naturally required to ensure that the mutual rendering is accurate. This is particularly important if the UE is integrated in AR / VR glasses, for example, because the user's movement affects the movement of the UE and the user's viewpoint. If two remote users wearing AR / VR glasses participate in a remote (virtual reality) application, the required communication flow rate can be kept low if their relative posture is stable, but if their relative posture changes, a higher flow rate may be required.

[0327] Relative posture can refer to: relative speed, relative position, relative acceleration, relative orientation, etc.

[0328] Therefore, in an embodiment, the transmitting TD(TE) and the receiving TD(TE) share their poses (position, velocity, acceleration, orientation, etc.) with each other. This allows them to determine their relative poses and therefore determine whether the requested data rate is sufficient.

[0329] In another embodiment, the transmitting TD(TE) and the receiving TD(TE) share their positions (position, velocity, acceleration, orientation) with a central management entity. This allows the management entity to determine the relative positions of the transmitting TD(TE) and the receiving TD(TE), and therefore determine whether the required data rate is sufficient.

[0330] In another embodiment, the TD(TE) / central management entity determines the data rate for a given application (eg, a virtual reality application) based on the relative positions of the transmitting TD(TE) and the receiving TD(TE).

[0331] In another embodiment, the telecommunication system may have an interface to share the TD's pose (position, velocity, acceleration, orientation, ...) and / or the TD's relative pose with external applications, and / or receive configuration regarding the required data rate from external applications.

[0332] In another embodiment, a device (eg, TD) or a management entity may run a prediction model to predict the poses of other TDs and / or predict relative poses with respect to other TDs in order to estimate the required data rate.

[0333] In another embodiment, the transmitting TD / receiving TD and / or the management entity uses the current or predicted relative pose to orchestrate / coordinate / trigger / adapt the following:

[0334] - resource allocation in the RAN of the sending UE and / or the receiving UE,

[0335] - the data rate of the coded data,

[0336] -…

[0337] In another embodiment, a device (eg, TD) or a management entity may perform one or more of the following tasks:

[0338] - Task 1: Requesting and / or receiving from the RAN or core network service (e.g. AMF, SMF, LMF, NWDAF) of the sending UE and / or receiving UE an indication of communication parameters (e.g. between two UEs or between two RANs) and / or (relative) location parameters of the UEs, and / or

[0339] - Task 2: Request and / or receive from the RAN or core network services (e.g. AMF, SMF, LMF, NWDAF) of the sending UE and / or receiving UE an indication of available resources in the RAN (e.g. which timeslots are available, which resources can be used to provide a given throughput with a given latency, which QoS / QoE is possible and / or can be guaranteed, which CPU power is available) and / or

[0340] - Task 3: The RAN of the sending UE and / or the receiving UE may be indicated with the required resources (e.g. which specific time slots are preferably allocated for the communication link between the UEs, which CPU resources are required, ...) and / or

[0341] - Task 4: The remote device may be requested to adapt the transmission quality (e.g. by adapting the data rate) and / or

[0342] - Task 5: may expose information to an external management function (eg, AF) and / or receive commands related to tasks 1-4 from the external management function.

[0343] -Task 6: Some of these tasks may be performed in a specific order, e.g., first Task 1, then Task 2, then Task 3, and / or

[0344] - Task 7: Perform some of these tasks (e.g., Task 3 and / or Task 4) upon determining a change in a parameter of Task 1 (e.g., communication parameters (higher latency) or relative pose (e.g., change in orientation of the UE)).

[0345] This may allow the resources allocated in the RAN of the sending UE and / or the receiving UE to be synchronized and sufficient to deliver the required service without incurring, for example, too high a latency.

[0346] Operation-QoS Balancing

[0347] In some virtual reality scenarios, a user may have different / better communication parameters than other users. Fig.11 As shown, user 2 is located between user 1 and user 3. Therefore, user 2 receives the communication stream from user 1 (or user 3) twice as fast as user 1 (or user 3) receives the communication stream from user 3 (or user 1) (with half the delay).

[0348] If the technology described in the above embodiments is applied, the communication streams originating from the TD and TE of each corresponding transmitting user can arrive at the receiving TD / TE of each user in a synchronous manner. This may also require the use of a prediction model. However, since user 2 is still closer to user 1 and user 3, the quality of the predicted signal may still be better for user 2 than for user 1 and user 3. The reason is that user 2 needs to apply the received input to a prediction model that predicts the less distant future; however, for example, user 1 needs to apply the asynchronous received input to a prediction model that requires a prediction of the more distant future.

[0349] In this case, the virtual reality application can instruct the (5G) TSM and the TSM underlying networking infrastructure to balance the provided QoS provided to the users involved in the virtual reality communication session. This may involve, for example:

[0350] First, penalize a user who may have a communication advantage due to, for example, his location, for example by delaying the communication flow originating from such a user (user 2 in the example), and / or

[0351] Second, provide a suboptimal model to a user (TE / TD) in a more favorable situation (User 2 in the example), and / or

[0352] Third, assigning more computing resources or more powerful models to some of the users in less favorable situations (User 1 and User 3 in the example), and / or

[0353] Fourth, configure the parameters of the above actions.

[0354] Although this may sound illogical, this may be useful in applications such as games, since otherwise such users would always have a given advantage over the rest of the users.

[0355] To achieve this goal, the virtual reality application can instruct the core networking infrastructure to provide QoS balancing. The core networking infrastructure then configures the communication flow, for example, by deploying policies at the edge networking infrastructure to enforce QoS balancing.

[0356] Operations with split rendering

[0357] The above embodiments also apply to architectures using split rendering. Split rendering means that the heavy rendering process is done by a device with high computing resources (e.g., a tactile edge device (TED), e.g., an edge server), and the subsequent user-specific or device-specific light rendering is done locally at, for example, a tactile device (TD). Split rendering allows the computation to be offloaded to the TED, thereby keeping the TD simple. When a split rendering architecture is used, the prediction model can be executed at the TED (e.g., as described in the embodiments related to model registration).

[0358] Each user can execute one or more prediction models. One of these prediction models can be, for example, a volumetric video (VV) representation for predicting a user so that the user can be represented in a realistic manner. The TED can execute multiple prediction models, such as a prediction model for each user, and may require synchronization of data streams for multiple users at different locations / TEs with multiple tactile sensors (e.g., based on embodiments related to synchronization of operation-data streams). If the TED runs prediction models for multiple remote users, the TED can render a combined, time-synchronized, and time-predicted representation of all users involved in the virtual reality session, such as a VV-rendered representation. Time synchronization means that the generated data streams are aligned, that is, they follow a common clock. Time-synchronized data streams received (from other remote TEs) may arrive later at a given time increment compared to the local clock of the local TE. Therefore, time prediction means that the representation is predicted as a time increment that will synchronize it with the local clock of the local TE / TED in the future. The time increment may depend on the latency or communication parameters between each pair of remote TE / TED.

[0359] TED may consume information about local users, such as local rendering devices (e.g., TDs) associated with users in a local environment. For example, TED may consume the height, position, and orientation of VR / AR glasses worn by a user. Using this information, TED may derive a TD-specific representation of the environment that can be consumed by the user's rendering device (TD). For example, from the perspective of the user's rendering device (e.g., VR / AR glasses), the representation may be a 2D representation of a volumetric video rendered at an edge server. For example, TED may use consumed / received data related to the pose of VR / AR glasses to determine the relative pose of VR / AR glasses relative to another user, and adapt communication parameters accordingly, such as the required bit rate.

[0360] In this environment, the local TED requires the communication system to allocate communication resources so that the TDs in the environment can continuously provide input related to, for example, their posture. This can be done in a time-deterministic manner. For example, the 5GS can allocate H resource blocks every m milliseconds to send data related to head posture. If this is done, the delay when the TED receives the user's posture is T=Tsensing+m+Tflight, where Tsensing is the processing delay from sensing the posture until the value can be sent, m is the delay due to discrete measurements, and Tflight is the propagation time from the TD to the TED. This may involve allocating deterministic uplink communication resources through a mechanism similar to, for example, semi-persistent scheduling, so that the TD can keep sending its input in a reliable and time-deterministic manner. This may require the TED to take T into account, for example, by running a prediction model of posture that allows predicting the actual current posture of the user given past samples.

[0361] Similarly, TED can ask the communication system to allocate communication resources so that TDs in the environment can continuously receive TD-specific representation input generated in TED. TED may also need to consider the transmission delay T = Trending + m + Tflight in the local TE, where Trending is the time required to update the rendering in the local TD once the data is received, m is the delay caused by the discrete transmission time, and Tflight is the propagation time from TED to TD.

[0362] In this embodiment, the latency of uplink communications (i.e., from TD to TED), including information about the TD (e.g., posture), and the latency of downlink communications and local rendering can be part of the communication parameters considered when synchronizing data streams from other users at other locations and / or applying predictive models.

[0363] System architecture enhancements for next generation real-time communications

[0364] TR 23.700-87v1.0.0 describes 5G system architecture enhancements for next-generation real-time communications, including:

[0365] • IMS network architecture enhancements required to support AR telephony for different types of AR-capable UEs.

[0366] Changes to IMS processes, including signaling and media processing, required to support AR telephony

[0367] Solutions #8 and #9 in TR 23.700-87 address these architectural enhancements. TR 23.700-87 concludes:

[0368] The data channel architecture is used as a baseline to support AR telephony communications.

[0369] If the UE requires network support for media rendering, then use the architecture and procedures specified in Solution #9.

[0370] Otherwise, if the UE can perform media rendering without network support, the process specified in Solution #8 is used as the baseline for the terminal rendering process.

[0371] In an embodiment, the system and functionality described in Solution #8 in TR 23.700-87 are extended to support some of the above embodiments. To this end, TR 23.700-87 Figure 6 .8.2-1 and the related communication flows between two UEs, including the following three processes: (1) IMS multimedia telephone call; (2) guided data channel (DC) establishment; and (3) application DC establishment, can be further improved through the embodiments in the present application.

[0372] The AR phone communication is exchanged via RTP. In one embodiment, these processes can be extended by:

[0373] Incorporate the capability to determine the end-to-end latency between UEs (e.g. during application DC establishment or during application data exchange), and / or

[0374] Incorporating capabilities for determining end-to-end parameters (e.g., relevant posture or communication parameters between UEs) (e.g., during application DC establishment or during application data exchange), and / or

[0375] Incorporating (e.g. once application DC setup is complete) deployment of a predictive model of the UE (or UE environment) on the peer side, and / or

[0376] • Incorporate capabilities for configuring / synchronizing communication flows (e.g., configuring QoS) and / or computing resources (e.g., allocating CPU / storage resources at edge servers) and / or policies (e.g., latency, sensor rates to apply) and / or compression algorithms.

[0377] In another embodiment, the system and functionality in solution #9 in TR 23.700-87 are extended to support some of the above embodiments. Figure 6 .9.2.2-1 describes the communication flow between two UEs with a network rendering process, where the augmented reality (AR) media processing network function (ARMF) is responsible for AR communication media transmission and media rendering functions, including the following functions:

[0378] AR rendering logic: Application-based rendering logic that controls AR communications.

[0379] AR media processing functions: including visual engine and 3D rendering engine. The visual engine and 3D rendering engine will build a spatial map and render scenes, virtual character models and 3D object models according to the field of view, posture, position, etc. sent from the UE using the data channel.

[0380] exist Figure 6 In .9.2.2-1, ARMF plays the role of TE, TED or TSM described in the above embodiments. The solution / architecture / protocol in TR23.700-87 can be extended to:

[0381] Incorporating the capability to determine end-to-end latency between UEs (e.g., during an AR media rendering negotiation process, or during an AR session media renegotiation process for network rendering, or during a network rendering process), and / or

[0382] Incorporating capabilities for determining end-to-end parameters (e.g., relevant posture or communication parameters) between UEs (e.g., during application DC establishment or during application data exchange), and / or

[0383] Incorporate the capability to synchronize incoming data streams for different remote users at different locations, specifically with respect to step 20 (sending AR media from UE-A to DCMF by applying DC) and step 21 (transferring AR media from DCMF to ARMF) or with respect to step 24 (sending rendered audio / video from UE-A to P-CSCF / IMS-

[0384] AGW) and step 25 (transfer rendered audio / video to ARMF), and / or

[0385] • Incorporate the deployment of a predictive model of the UE (or UE environment) at the ARMF.

[0386] Incorporate capabilities for configuring / synchronizing communication flows (e.g., configuring QoS) and / or computing resources (e.g., allocating CPU / storage resources at edge servers) and / or policies (e.g., delays, sensor rates to apply) and / or compression algorithms.

[0387] In another embodiment, the UE or the ARMF responsible for the UE UE1 synchronizes the incoming data streams (e.g., DSi received at UE1 from UEi, where i=2,...,N). The synchronization is achieved by: determining the delay Di of each DSi; determining the DSj that arrives with the highest delay Dj; delaying the data stream DSi by a certain amount Dj-Di, where the delay can be performed by buffering DSi; and optionally passing all synchronized data streams to the corresponding prediction models PMi, where all PMi are configured to perform predictions for the same delay. Each PMi takes the (delayed) DSj as input and calculates a predicted value. This embodiment represents a distributed approach, where each UE performs the above-mentioned synchronization (similarly, each UE is linked to an ARMF that performs such synchronization and / or prediction on behalf of the UE). This embodiment applies QoS balancing because all PMs predict the output based on the same delay Dj.

[0388] In another embodiment, instead of the previous embodiment, the data stream DSi may not be synchronized with other DSs and may be passed to PMi, where PMi is configured to perform prediction for delay Di (instead of Dj). This embodiment applies QoS balancing. This embodiment does not apply QoS balancing because PMi predicts the output based on delay Di (where i=1, ..., N).

[0389] In another embodiment, a centralized network entity (e.g., an application server (AS), such as a centralized ARMF or an ARMF in a home network, or such as in TR 23.700-87) Figure 6 .9.1.3-1) is responsible for performing such synchronization and / or prediction of the output on behalf of the UEs involved. In this embodiment, it is considered that the (communication) parameters between UEi and AS may be specific to UEi, for example, the distance between UEi and AS (and therefore the minimum latency / delay Di) is specific to UEi. The AS may receive data streams of UEi (i=1,…,N) with delay Di, and the AS may synchronize them, for example, as shown in the previous embodiments. The AS may then apply the (optionally synchronized) received data streams to a prediction model PMi for predicting the output (e.g. Oi) further in the future at least at time Di, so that when Oi is delivered at UEi, Oi is delivered on time based on the communication delay Di between AS and UEi.

[0390] In another embodiment, the UE may perform synchronization and / or prediction of the output on behalf of the UE itself involved, whereby the UE may be configured by the ARMF or other network function or by the AR application server.

[0391] Example use cases

[0392] The above-described embodiments may support multiple use cases.

[0393] The first use case is about implementing a real-time teleconferencing service.

[0394] In this first use case, three users are using 5GS to join an immersive VR conference call. Users Bob, Lukas, and Yong are located in the United States, Germany, and China, respectively. Each of the users is served by a local VR edge computing server (MECS) hosted in 5GS, and each of the servers is located near the users it serves. When a user joins a VR conference call, the user's avatar is loaded into the VR edge computing servers of other users. For example, the VR edge computing server close to Bob hosts the avatars of Yong and Lukas.

[0395] A minimum communication latency is determined at large distances between users (e.g., the distance between the United States and China is about 11640 km), e.g., 11640 / c=38 ms. This latency may also be variable due to a variety of reasons, such as, for example, congestion or delays introduced by (variable processing times of) hardware components such as sensors or rendering devices. Since this value may be too high and variable for a truly immersive virtual reality teleconferencing experience, each of the deployed avatars includes one or more predictive models of the person it represents, and the predictive models allow for rendering a synchronized predicted (current) avatar representation of the remote user in a local edge server.

[0396] Fig.12 This exemplary scenario is shown where the MECS at location 3 (USA) runs the prediction model for the remote users (Yong and Lukas) and takes as input the sensory data received from all users (Yong, Lukas, and Bob) and generates synchronized predicted (current) avatar representations of the users to be rendered in Bob's local rendering device.

[0397] In this first use case, the following preconditions and assumptions apply to this use case:

[0398] 1. Up to three different MNOs operate 5GS for providing VR teleconferencing services.

[0399] 2. Users Bob, Lukas and Yong have subscribed to the virtual reality teleconferencing service.

[0400] 3. Each of the users (eg, Bob) decides to join the teleconference session.

[0401] In this first use case, the following service flows need to be provided:

[0402] 1. Each of the users (eg, Bob) decides to join the teleconference session and agrees to deploy their avatar.

[0403] 2. The virtual reality sensor at each user senses the real-time recording of each of the users. The real-time representation of each of the sensed users is distributed to the virtual reality edge computing servers of other users in the virtual reality teleconference session.

[0404] 3. Each of the virtual reality edge computing servers applies the incoming data stream representing each of the remotely located users to the corresponding avatar prediction model (which takes into account current communication parameters / performance (e.g., latency)) to create a combined, synchronized, and current representation of the remote users to be provided as input to the rendering device in the local TE.

[0405] In this first use case, the main post-condition is that each of the users enjoys an immersive virtual reality conference call.

[0406] In this first use case, there are new requirements needed to support the use case, namely, the 5G system should provide means for synchronizing the data streams of multiple virtual reality (sensor and rendering) devices associated with different users at different locations. The second requirement refers to the need to support the distribution and execution of predictive models in edge servers.

[0407] The second use case is about implementing a real-time teleconferencing service.

[0408] In this second use case, three users are using 5GS to join an immersive VR conference call. Users Bob, Lukas, and Yong are located in the United States, Germany, and China, respectively. Each of the users is served by a local VR edge computing server hosted in 5GS, and each of the servers is located near the users it serves. When the users join the VR conference call, the avatars of each of the users are loaded into the VR edge computing servers of the other users. For example, the VR edge computing server close to Bob hosts the avatars of Yong and Lukas.

[0409] The minimum communication delay is determined at large distances between users (e.g., the distance between the United States and China is about 11640 km), for example, 11640 / c=38 ms. This delay may also be variable due to a variety of reasons, such as, for example, congestion or delays introduced by hardware components such as sensors or displays. Since this value may be too high for a truly immersive virtual reality teleconferencing experience, each of the deployed avatars includes one or more predictive models of the person it represents, and the predictive models allow the predicted (current) representation of the person to be rendered in the edge server.

[0410] Each of the virtual reality edge computing servers can combine data streams from other users and create a meaningful real-time representation of the user. Since the user's virtual reality rendering device has limited rendering capabilities, a split rendering approach is applied, in which the virtual reality edge computing server performs the heavy rendering load and distributes personalized views to the user's virtual reality rendering device.

[0411] In this second use case, the following preconditions and assumptions apply to this use case:

[0412] 1. Up to three different MNOs operate 5GS for providing VR teleconferencing services.

[0413] 2. Users Bob, Lukas and Yong have subscribed to the virtual reality teleconferencing service.

[0414] 3. Each of the users (eg, Bob) decides to join the teleconference session.

[0415] In this second use case, the following service flows need to be provided:

[0416] 1. Each of the users (eg, Bob) decides to join the teleconference session and agrees to deploy their avatar.

[0417] 2. The virtual reality sensor at each user senses the real-time recording of each of the users. The real-time representation of each of the sensed users is distributed to the virtual reality edge computing servers of other users in the virtual reality teleconference session.

[0418] 3. Each of the virtual reality edge computing servers applies the real-time data stream representing each of the remotely located users to the corresponding avatar prediction model (which takes into account current communication parameters / performance (e.g., latency)) to render a combined, synchronized, and current representation of the remote users.

[0419] 4. Each virtual reality edge computing server in the virtual reality edge computing server keeps collecting the current posture of the local user (for example, through virtual reality sensors around the user), such as the position in his / her room, the height and orientation of his head, etc.

[0420] 5. Each of the virtual reality edge computing servers maintains processing of the following:

[0421] (1) Combination, synchronization, and current representation of the remote user's rendering, taking into account

[0422] (2) Current posture of the local user

[0423] A view of the remote user is derived for the local user, and the view can be distributed to the virtual reality rendering device of the local user.

[0424] In this second use case, the main post-condition is that each of the users enjoys an immersive virtual reality conference call.

[0425] In this second use case, there is also a new requirement needed to support the use case, namely, the 5G system should provide a means for synchronizing the data streams of multiple virtual reality (sensor and rendering) devices associated with different users at different locations. The second requirement refers to the need to support the distribution and execution of prediction models in edge servers.

[0426] Therefore, a method and a corresponding edge server configured to perform such a method may be defined, wherein the method comprises:

[0427] Receiving the prediction model at the edge server,

[0428] receiving data from a remote user device and / or a haptic device in the area of ​​interest,

[0429] For example, by inferring the missing samples, rendering the rendered data based on the received data,

[0430] The rendered data is sent to the destination user device.

[0431] General Notes

[0432] Although the present invention is described in the context of a virtual space, such as a virtual reality, its application is not limited to this type of operation. Low-latency systems such as those in industrial IoT systems will also benefit from the teachings of the present invention and its embodiments.

[0433] By studying the drawings, the disclosure and the appended claims, those skilled in the art can understand and implement other variations of the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit can implement the functions of several items recorded in the claims. The fact that certain measures are recorded in mutually different dependent claims does not indicate that the combination of these measures cannot be used advantageously. The foregoing description details certain embodiments of the present invention. However, it should be understood that, regardless of the details in which the foregoing appears in the text, the present invention can be practiced in many ways and is therefore not limited to the disclosed embodiments. It should be noted that when describing certain features or aspects of the present invention, the use of a particular term should not be regarded as implying that the term is redefined herein as being limited to including any particular characteristics of the features or aspects of the present invention associated with the term. In addition, the expression "at least one of A, B and C" should be understood as separate, i.e., "A and / or B and / or C".

[0434] A single unit or device may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0435] The operations described similar to those indicated in the above embodiments can be implemented as program code units of a computer program and / or dedicated hardware of related network devices or functions, respectively. The computer program can be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium, provided together with other hardware or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

Claims

1. An apparatus for providing a synchronous input to a third device, the apparatus comprising: a memory for storing communication parameters shared between the third device and the first device, and / or between the third device and the second device, b. a communication unit for receiving a first communication stream from said first device and / or a second communication stream from said second device, Therein, the communication flow is synchronized based on the communication parameters.

2. The device according to claim 1, wherein: The communication parameters include: - the distance between the third device and the first device and / or the distance between the third device and the second device, or - a delay from the first device to the third device and / or a delay from the second device to the third device, or - other communication parameters shared between the first device and the third device and / or between the second device and the third device.

3. The apparatus according to any one of the preceding claims, further comprising a computing unit for executing a prediction model, the prediction model taking the communication parameters between the first device and the third device as input to predict a control input for the third device, wherein The control input includes a predicted communication parameter between the first device and the third device.

4. The apparatus according to any of the preceding claims, further comprising a computing unit for executing a prediction model, the prediction model taking as input at least the communication flow of the first device to predict a control input for the third device.

5. The device according to claim 4, wherein: The prediction model is at least one of the following: - a model derived from a common prediction model based on parameters shared between at least said first device and said third device, -Generate models.

6. A device according to any one of the preceding claims, wherein: The communication parameters are obtained by running a protocol with the first device.

7. A device according to any one of the preceding claims, wherein: The communication parameters are configured by a management device.

8. A device according to any one of the preceding claims, wherein: The communication parameter can be at least one of the following: - a delay between the third device and the first device and / or the second device, -QoS, - a distance between the third device and the first device and / or the second device, - the computational requirements for processing said communications, - computing power to process said communications, - the memory requirements for processing said communications, - memory capacity for processing said communications, - available bitrate, - the number of communicating parties, - a (relative) pose relative to the third device and the first device and / or the second device, - a (relative) speed relative to said third device and said first device and / or said second device, - a (relative) acceleration relative to said third device and said first device and / or said second device, - a (relative) rotation relative to said third device and said first device and / or said second device.

9. A system comprising at least one third device and at least one remote first device, said third device comprising the apparatus as claimed in claims 1-8, said at least one remote first device being adapted to transmit a communication stream received at said third device.

10. A method for providing synchronized input to a third device, the method: a. storing in a memory the communication parameters shared between the third device and the first device, and / or between the third device and the second device, b. receiving, by a communication unit, a first communication stream from the first device, and / or a second communication stream from the second device, c. Synchronizing the communication flow based on the communication parameters.

11. An apparatus for using a prediction model of a first device, the apparatus comprising: a. a storage unit for storing the prediction model of the first device, b. a communication unit for acquiring a communication parameter characteristic of a communication link between the first device and the third device, c. a computing unit capable of using said prediction model of said first device, The output of the prediction model is obtained based on the prediction model and the communication parameters.

12. The device according to claim 11, wherein The output of the predictive model enables a synchronous communication flow between the first device and the third device.

13. The device according to claim 11 or 12, wherein: The output of the predictive model is a derived predictive model, and wherein a required number of input parameters in the derived predictive model is less than a required number of input parameters of the predictive model.

14. A method of using a prediction model of a first device, the method being adapted to: a. storing the prediction model of the first device, b. obtaining communication parameter characteristics of the communication link between the first device and the third device, c. Using the prediction model of the first device, based on that the output of the prediction model is obtained based on the prediction model and the communication parameters.

15. A computer program product comprising code means for producing the steps of claim 10 or 14 when run on a computer device.

Images