System and method for multi-modal adaptive spatial remote communication

Abstract

Systems and methods for adaptive spatial remote communication are provided. A method includes determining a requirement parameter for a first system, receiving a capability parameter for a second system that is used by a second user in communication with a first user of the first system, comparing the requirement parameter and the capability parameter. Based on the comparison, the method further includes determining whether to adapt one or more content representations for one or more of the first system and the second system. In response to determining that at least one of the content representations is to be adapted, the method causes at least one of the first system and the second system to adapt the at least one of the content representations to be presented, wherein the adapting is based on a first modality and / or a first quality level.

Description

SYSTEM AND METHOD FOR MULTI-MODAL ADAPTIVE SPATIAL REMOTE COMMUNICATION RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 534,995, filed August 28, 2023, the disclosure of which is hereby incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present disclosure relates generally to spatial remote communication. BACKGROUND

[0003] A key challenge in the development of technologies for remote communication is recreating the rich spatial and immersive nature of in-person interactions. The ability to share three-dimensional information about people and their surroundings, and to view and interact with said information spatially, has only recently become realizable with the development of advanced spatial computing devices such as Augmented Reality and Virtual Reality headsets, tools for environment capture such as depth cameras, and high bandwidth 5G networks capable of streaming large amounts of volumetric data.

[0004] Using a combination of these technologies, it is possible to stream 3D representations of self and surroundings for real-time remote collaboration. These 3D representations are commonly captured and streamed in the form of point clouds and meshes, requiring data transfer speeds upwards of 1 Gbps if used in their raw, uncompressed form. As widely available networks normally have far lesser bandwidth and greater network uncertainty (latency, jitter, packet loss), researchers have developed mechanisms to optimize the streaming of 3D information for volumetric remote communication.

[0005] One approach of optimizing streaming of 3D information includes methods for progressive encoding and decoding of volumetric video and point clouds based on tiling and multi-scale encoding of video streams. In addition, as the viewers of these 3D information streams are often free to move and look around relative to the shared environment, some approaches consider the viewer’s position and viewport to determine objects of interest, and correspondingly encode the environment at varying levels of detail and quality. This knowledge of relative pose can be used to derive perceived quality estimates based on models of visual acuity.

[0006] Other approaches have implemented the offloading of computationally intensive operations such as transcoding and rendering to servers on the network edge, thereby reducing the processing demand when viewing spatial information on mobile devices. Using a combination of such methods, it has been demonstrated that volumetric remote communication can be more efficiently achieved over commercial networks.

[0007] Further, ongoing efforts towards the standardization of 3D streaming are taking place at Moving Picture Experts Group (MPEG) (3D Graphics Group (3DG) and Point Cloud Compression (PCC), also MPEG-I (ISO / IEC 23090)), and International Telecommunication Union (ITU) and Video Quality Experts Group (VQEG) are currently working towards establishing test procedures to assess user’s quality of experience during immersive remote communication (ITU-T P.1320).

[0008] However, the existing approaches have certain challenge(s) and disadvantages. When placed under network constraints, existing technologies and approaches primarily optimize for visual perception of quality within fixed representation modalities (mesh, point cloud). While this approach widens the allowable window of network degradation, the lower bound may still not be met by commercial networks. Additionally, these approaches are not optimized for all devices.

[0009] In such situations, existing approaches either fail to provide the minimum quality of experience required to support interaction or recommend that users fall back to using conventional methods or systems of remote communication which are less bandwidth intensive, such as video or voice calls. However, the existing approaches do not provide mechanisms to seamlessly and gracefully move between such methods of representation in the same system.

[0010] Parallel research in the fields of avatar interaction has demonstrated that realism is not always the goal, and that stylized representations of users offer comparable user experience in certain task contexts. Research and commercial prototypes have also demonstrated that expressive user representations can be created from sparse sensor and tracking data, requiring far less bandwidth to transmit in real time. However, these alternate representations exist in siloed applications, and not as part of a single system with a suite of possible user representations given network, task, and device constraints.

[0011] Furthermore, users may not always want to share the highest-resolution volumetric representations of themselves, but still give a sense of presence and interaction (to preserve privacy). In group settings, it might be overwhelming to view high-resolution point cloud representations of all remote collaborators, and the option to view key actors in higher-fidelity modalities, and background participants in lower-fidelity, does not yet exist in systems today.SUMMARY

[0012] Systems, such as a host or server, and systems for adaptive spatial remote communication are provided.

[0013] An exemplary method includes determining a requirement parameter for a first user system, receiving a capability parameter for a second user system that is used by a second user in remote communication with a first user of the first user system, and comparing the requirement parameter and the capability parameter. The method further includes, based on a result of the comparison of the requirement and the capability parameter, determining whether to adapt one or more content representations for one or more of the first user system and the second user system, and in response to determining that at least one of the content representations is to be adapted, causing at least one of the first user system and the second user system to adapt the at least one of the content representations to be presented to at least one of the first user and the second user, wherein the adapting is based on a first modality and / or a first quality level.

[0014] Embodiments of the method further includes one or more of the following features. The determining whether to adapt one or more content representations of one or more of the first user system and the second user system may include determining that the capability parameter for the second user system is greater than the requirement parameter for the first user system. The causing at least one of the first user system and the second user system to adapt the at least one of the content representations may include negotiating, based at least in part on the capability parameter for the second user system, a content representation for streaming and receiving media content from the second user system. The causing at least one of the first user system and the second user system to adapt the at least one of the content representations may include receiving, from the second user system, media content at the negotiated content representation.

[0015] Additional embodiments include one or more of the following features. Causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes determining, based at least in part on the requirement parameter for the first user system, how much of the media content is to be rendered. Determining whether to adapt one or more content representations of one or more of the first user system and the second user system includes determining the capability parameter of the second user system is less than the requirement parameter of the first user system. The causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes receiving, from the second user system, media content at a content representation that is associated with the capability parameter for the second user system, and determining, based on a task of the remote communication between the first user and the second user, at least one of amodality and a quality level for the media content that provides a similar Quality of Experience than a Quality of Experience obtained when the second user system has a capability parameter that satisfies the requirement parameter of the first user system.

[0016] The determining whether to adapt one or more content representations of one or more of the first user system and the second user system includes determining that the capability parameter of the second user system and the requirement parameter of the first user system are comparable and fluctuating. The causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes negotiating, based at least in part on the capability parameter for the second user system, a content representation for streaming and receiving media content from the second user system, wherein the content representation is associated with a highest capability parameter for the second user system.

[0017] Embodiments of the method further include receiving an updated capability parameter for the second user system, comparing the requirement parameter with the updated capability parameter, based on a result of the comparison of the requirement and the updated capability parameter, determining whether to adapt the content representations of one or more of the first user system and the second user system, and in response to determining that at least one of the content representations is to be adapted, causing at least one of the first user system and the second user system to adapt the at least one of the content representations to be presented to at least one of the first user and the second user, wherein the adapting is performed based on a second modality and / or second quality level that is different from the first modality and / or first level quality. Hosts or servers configured to perform the operations of this method are also included.

[0018] The methods and hosts or servers provide improvements in adaptive spatial remote communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawing figures incorporated in and forming a part of this specification illustrate several aspects of the disclosure, and together with the description serve to explain the principles of the disclosure.

[0020] Figure 1 illustrates a block diagram of an exemplary system for multi-modal adaptive spatial remote communication, in accordance with some embodiments.

[0021] Figure 2 illustrates a flow diagram of exemplary operations that can be performed by one or more of the adaptive spatial experience modulator (ASEM), in accordance with some embodiments.

[0022] Figure 3 illustrates a block diagram of exemplary representation of requirements and capabilities that can be determined at an ASEM, in accordance with some embodiments.

[0023] Figure 4 illustrates exemplary representations with different content modalities, according to some embodiments of the current disclosure.

[0024] Figure 5 illustrates an exemplary flow of operations that can be performed in an ASEM for modulating content representation for a user system, according to some embodiments of the current disclosure.

[0025] Figure 6 is a block diagram of a communication system in accordance with some embodiments.

[0026] Figure 7 is a block diagram a UE in accordance with some embodiments.

[0027] Figure 8 is a block diagram a network node in accordance with some embodiments.

[0028] Figure 9 is a block diagram of a host, , in accordance with various aspects described herein.

[0029] Figure 10 is a block diagram illustrating a virtualization environment in which functions implemented by some embodiments may be virtualized.

[0030] Figure 11 is a communication diagram of a host communicating via a network node with a UE over a partially wireless connection in accordance with some embodiments. DETAILED DESCRIPTION

[0031] The embodiments set forth below represent information to enable those skilled in the art to practice the embodiments and illustrate the best mode of practicing the embodiments. Upon reading the following description in light of the accompanying drawing figures, those skilled in the art will understand the concepts of the disclosure and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts and applications fall within the scope of the disclosure.

[0032] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. Embodiments herein describe dynamic and adaptive systems and methods for modulating the content and representations of self and surroundings in shared spatial remote communication. This system can function as a standalone remote communication platform or be integrated into any other remote communication platform (e.g., via an application programming interface (API). In some embodiments, information about current and predicted network parameters, task needs, device capabilities, and user experience is monitored and aggregated. The information is analyzed in real time before computing and providing an optimized set of user and content representations to be selected and ordered from the full breadth of the fidelity and quality spectrum. These representations are then either modulated seamlesslyand gracefully as the monitored parameters change or the users are offered the option to choose from a set of optimal representations for a given context.

[0033] The embodiments herein present one or more of the following features. For example, some embodiments present an Adaptive Spatial Experience Modulator (ASEM). The adaptive spatial experience modulator is a network node or host capable of running on both client and server-side. The ASEM monitors and analyzes the state of remote communication across channels relating to network status, task needs, device constraints, and user experience, and modulates the representation of user and environment representations to provide an optimal holistic quality of experience. The ASEM can be used as a standalone application for remote communication, or function as an external API, potentially integrating into any remote communication platform.

[0034] Some embodiments herein present a Dynamic Spatial Content Representations (also referred to as content representation), a novel application-layer feature that comprises a continuum of possible visual and functional representations pertaining to users and environments and is capable of seamlessly transitioning between points on the continuum based on information provided by the ASEM, either directly or mediated by user input.

[0035] Some embodiments present a method for multi-modal adaptive spatial remote communication. The method can include establishing a connection for spatial remote communication between two or more user devices of one or more users. The user devices may be the same or different. Additionally or alternatively, the communication is established with interfaces with existing remote communication channels via an API. The method can further include initializing the respective dynamic spatial content representations at a default optimal state. The method can further include continuously monitoring various information regarding the connection. For example, the information can include network information for each remote user (e.g., network parameters (uplink and downlink capabilities), and / or predicted anticipated changes to these network parameters for a given time window). Additionally, the information can include one or more parameters indicative of the experience. These parameters may include a type of task that is to be performed by one or more of the users (e.g., interview, remote musical collaboration, machine assembly, physical therapy, remote learning, meeting, etc.), an indication of the environment, and interaction parameters relating to the ongoing communication between users. The information can further include device capabilities (e.g., device capabilities in terms of capture, display, and processing). The information can further include modular additions (e.g., each channel can also include additional models to enhance the assessment of current system status (visual quality, acuity estimations)). The method can further include determining based on the information an optimal set of possible representations for spatial content in themedia stream. The set of possible representation for spatial content can pertain to self and surrounding environments. Based on the resulting set of representations, seamless transitions between possible modalities and levels of detail of the respective dynamic spatial content representations is directed. Additionally, or alternatively, one or more of the users is provided with the ability to modulate (e.g., through an interface) currently active representations and their level of detail. The method further enables the secure monitoring of user interactions and communication behavior to dynamically tune the adaptation algorithm for the specific set of users and contexts. In some embodiments, the method preserves the optimal algorithm for use in future interactions upon disconnection.

[0036] Certain embodiments may provide one or more of the following technical advantages. The methods and systems presented herein optimize content representation for a comprehensive measure of Quality of Experience. The described embodiments optimize for more than just visual perception. For example, embodiments herein consider factors such as the task at hand during the communication, and the nature of communication. The embodiments can also leverage the potential of digital spatial communication by offering a wide range of possible representations along the continuum of real and virtual, low and high fidelity – in contrast to prior existing approaches that merely modulates quality of a fixed representation.

[0037] The methods and systems presented herein provide great tolerance to network degradation. By including a wider range of representations, requiring bandwidths from as high as ~1 Gbps to ~1 Mbps and less, embodiments herein can operate under harsher network constraints, and still provide comparable levels of quality of experience.

[0038] The methods and systems presented herein are suitable for a wide range of contexts. The ability to seamlessly modulate between multiple options within the dynamic spatial content representation enables the preservation of user privacy and the ability to modify the level of social and representational detail in group interactions.

[0039] The methods and systems presented herein enable cross-application integration. By having the content representations be controlled by the ASEM, the system enables users to maintain consistent representations across various platforms, while maintaining a high overall quality of experience.

[0040] The methods and systems presented herein enable cross-device integration. The tightly coupled nature of the content representations and the ASEM enables users to transition between devices mid-interaction, and still maintain a cohesive and optimal experience based on the ideal representations for a given device modality.

[0041] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0042] Figure 1 illustrates a block diagram of an exemplary system for multi-modal adaptive spatial remote communication, in accordance with some embodiments. System 100 includes user system 110A and user system 110B coupled through a network 105. While system 100 is described as including two user systems, the embodiments herein are not so limited and system 100 may include any number of user systems coupled through a network 105. In one exemplary embodiment, system 100 enables two users of the respective user systems 110A-B to communicate with one another remotely. In a non-limiting example, the two users collaborate with each other remotely. For example, user of user system 110A is at a mechanical workbench and is demonstrating how to assemble a series of physical components to user of user devices 110B, who is a remote apprentice. Other examples include users during an interview process, users performing remote telemedicine (such as physiotherapy, etc.), users engaging in a remote musical collaboration. Other exemplary scenarios can be contemplated without departing from the scope of the present embodiments. While the embodiments herein describe exemplary system that are used for live communication and collaboration between users, some embodiments can be contemplated where a user is viewing a pre-recorded media content that can be adapted based on its local requirements and system capabilities.

[0043] The user system 110A includes display and interaction device(s). The display and interaction device(s) may include one or more devices that allow the user to view remote content media and enables spatial interaction in Augmented / Virtual Reality. These devices might include headsets, glasses, handheld displays, projection displays, as well as controllers and other peripheral devices or features which enable the viewing of and interaction with media received from the remote user system 110B. User system 110A further includes capture and tracking devices. The capture and tracking device are operative to enable tracking the user and object pose in space and to capture 3D information about the user and their environment. These devices might include motion-tracking sensors and systems, depth cameras, and other capture and tracking devices. The system 110A is further operative to process and transmit captured content, to process and display media received from the remote user system 110B, and any other operations for enabling communication between the two user systems. In a non-limiting example, the user system 110A includes an Augmented Reality (AR) headset and a series of depth cameras arranged in the room to capture information about themself and the components on the workbench.

[0044] The user system 110B includes display and interaction device(s). The display and interaction device(s) may include one or more devices that allow the user to view remote content media and enables spatial interaction in Augmented / Virtual Reality. These devices might include headsets, glasses, handheld displays, projection displays, as well as controllers and other peripheral devices or features which enable the viewing of and interaction with media received from the remote user system 110A. The user system 110B further includes capture and tracking devices. The capture and tracking device are operative to enable tracking the user and object pose in space and to capture 3D information about the user and their environment. These devices might include motion-tracking sensors and systems, depth cameras, and other capture and tracking devices. User system 110B is further operative to process and transmit captured content, to process and display media received from the remote user system 110A, and any other operations for enabling communication between the two user systems. In a non-limiting example, the user of the user system 110B uses a handheld mobile device running in AR mode and has a depth camera mounted on the mobile device to capture information about their upper- body and face.

[0045] In some embodiments, a user system can be comprised of a single device (e.g., a handled device) that is operative to capture data, transmit data, and display received content. Alternatively, a user system can be comprised of multiple devices, where each device being operative to perform one or more of the capture, the transmission, and the display of the received content (e.g., a handheld device, head mounted device, separate cameras, etc.).

[0046] The user system 110A is coupled with the system 110B through network 105. Network 105 can be used to transmit data between the two user systems as described in further details with respect to Figure 6. Network 105 is capable of transmitting processed information across user equipments included in the user systems and between the various adaptive spatial experience modulators and user systems.

[0047] The system 100 further includes one or more adaptive spatial experience modulators (ASEM) 120A-120B. In some embodiments, the system may include an ASEM associated with each user, such that the ASEM is operative to adapt the content representation received from a remote user system. Additionally or alternatively, the system may include an ASEM that is associated with more than one user, such that an ASEM can be used to adapt the content representation for more than one user. Each of ASEM 120A-B is operative to enable adaptive spatial communication between the two user systems. Each of the ASEM 120A-B can be implemented on a device of the user systems 110A-B or a computing device that is remote from both of the user systems (e.g., remote server, remote computer, remote laptop, etc.). In a non-limiting example of Figure 1, each of the user systems include a respective ASEM 120A-B running in standalone mode on a respective user device (AR headset, mobile device).

[0048] Figure 2 illustrates a flow diagram of exemplary operations that can be performed by one or more of the ASEMs, in accordance with some embodiments. The operations herein will be described with respect to ASEM 120A, however, similar operations are performed by ASEM 120B. At operation 210, ASEM 120A initializes communication with the user systems 110A-B through network 105. ASEM 120A receives information from the user systems 110A and processes the information to localize user and environment information. In some embodiments, ASEM 120A determines the maximum capability of content streaming for a given configuration of the associated user system 110A, in order to determine the maximum fidelity of the corresponding remote content. Content fidelity refers to the ability of the user system to render the content received from the remote user system in a perceptually similar way to the originally transmitted content. ASEM 120A determines the ideal content fidelity of remote content, based on one or more parameters such as task conditions, network conditions, device constraints, and quality of experience. In some embodiments, the initialization of the communication between the user systems 110A-B further includes specifying a common identifier (call ID or room name) and creating packets to exchange via handshake to join a bidirectional call. Once the communication is initiated, ASEM 120A begins sending and requesting information about users and their environments as described below. Similar operations can be performed by ASEM 120B. The flow of operations then moves to operation 220.

[0049] At operation 220, ASEM 120A determines capabilities and requirements for the user system 110A. In some embodiments, determining capabilities and requirements includes monitoring and processing one or more data streams collected from the device(s) of the user system 110A and network links or devices coupled with the user system 110A. Determining the capabilities can include capturing capabilities of one or more of the devices of the user system 110A in terms of capturing and processing media content that is to be transmitted to one or more remote user systems as well as the capabilities of the uplink network connection of user system 110A with one or more remote user systems.

[0050] Determining the requirements includes determining one or more parameter values that indicate how content received from remote user systems will be displayed in user system 110A. In one embodiment, determining the requirements includes determining a task that is performed by a user of the user system 110A and user focus for that task. Determining the requirements can further include the constraints and characteristics of the display device; objective human perception limits; a desired Quality of Experience (QoE); and downlinknetwork capabilities. In some embodiments, one or more of the requirements can be selected by the user of user system 110A (e.g., task, focus). In other embodiments, the requirements can be automatically determined by ASEM 120A. The flow of operations then moves to operation 230.

[0051] At operation 230, ASEM 120A performs adaptive media content communication based at least in part on the capabilities and requirements of the user systems. In one embodiment, performing adaptive media content communication can include setting an initial representation of the content of user systems 110A and environment representation of the content of user system 110B. The initial representation can be selected by the user of user system 110A. Alternatively or additionally the initial representation can be automatically determined by ASEM 120A based on the initial requirements and capabilities of user system 110A and / or user system 110B.

[0052] Performing the adaptive media content communication can further include, ASEM 120A transmitting information regarding current and / or predicted capabilities, as well as current and / or predicted requirements to one or more user systems, such as user system 110B. In some embodiments, transmitting the information can include creating and transmitting packets that include the information through network 105. In some embodiments, ASEM 120B further receives information from one or more user systems, e.g., ASEM 120B that includes current and / or predicted requirements and current and / or predicted capabilities of the one or more user systems. In some embodiments, the exchange of information between ASEMs is performed at a fixed cadence to facilitate persistent monitoring of the status of each of the user systems and enable adaptation to change that occur in a system.

[0053] ASEM 120A processes user system 110A requirements and the media content received from ASEM 120B to determine the nature of adaptation and modulation of the content representation that can be performed. The modulation can be automatic, where ASEM 120A seamlessly transitions between levels of detail and modalities of content representations based on changes in capabilities and requirements of the user systems. Additionally or alternatively, the modulation can be performed via user input, where the local user is presented with options and can specify which of the possible content representations they wish to view and interact with. The adaptation and modulation of the content representation is described in further detail below with respect to some embodiments. The adaptation and modulation take place at both locations (ASEM 120A-B), thereby continually adapting the content representations for user systems 110A-B.

[0054] Figure 3 illustrates a block diagram of exemplary representation of requirements and capabilities that can be determined at an ASEM, in accordance with some embodiments. In some embodiments, communication between two user systems involves users viewing andinteracting with three distinct types of content: representation of faces (F) (representing facial expressions and cues), representation of bodies (B) (representing body gestures, movement, physical operations), and representation of environment (E) (representing context-specific information such as objects and surroundings). ASEM 120A and ASEM 120B are operative to determine requirements and capabilities for each of their respective user systems 110A-B. The embodiments herein will be described with respect to ASEM 120A, however similar operations are performed by ASEM 120B to obtain corresponding requirement and capabilities.

[0055] In some embodiments, a requirement can be expressed as a vector (F, B, E) where the value of each of component ranges from 0 (not present) to 1 (to be present in highest fidelity). For example, a net requirement vector of (0, 0, 0) can correspond to an audio-only call, whereas a vector of (1, 1, 1) can correspond to a fully volumetric call (face, body, and environment streamed in highest fidelity 3D representation, usually a mesh). As the vector moves from 0 to 1 along any of the components, the associated requirement changes across both modality type, and the level of detail within each modality. A modality type can include mesh, point cloud, depth image, video, 3D avatar, or 2D avatar. Each modality can have one or more levels of details. A requirement for a content can include several requirement parameters. For example, the requirement can include context requirement, focus requirement, and / or device requirement. A context requirement enables varying levels of importance to different types of content (e.g., F, B, or E), based on the nature of interaction. For example, face-to-face conversations around a table involving negotiation or discussion may need high quality user representations of faces, but body and environment information will be less important. Thus, a context requirement vector would be represented with a high value (e.g., closer to 1) for the face element, and low values of each of the body and environment elements (e.g., closer to 0). In contrast, a remote walk-through of a virtual environment will require more information about user position, gestures, and surrounding context, with a relatively lesser emphasis on facial information. The context requirement vector values would also change during the interaction between the users, as different tasks can be part of the same communication thread (e.g., a face- to-face conversation which can be followed by a presentation where the surroundings or environment can become more important).

[0056] The requirement can further include a focus requirement. The focus requirement is indicative of how the user interacts with the received content and where he is likely to focus its gaze and attention. While a local user is interacting with remote content, their focus constantly changes based on the nature of the task. The local user may move closer or further away from the content, look towards or away from it. Thus, the amount and perspective of content in thelocal user’s viewport changes constantly. This influences the amount and resolution of content that is to be rendered.

[0057] The requirement can further include a device requirement. Even if the remote user streams high-fidelity content, not all displays can render full volumetric content, and the perception of visual stimuli varies significantly based on the display type and properties. Thus, depending on the nature of the local display, and based on prior assessments of perceived quality and visual acuity estimates, the fidelity of the requested representation can be further modulated and optimized.

[0058] ASEM 120A determines an ideal requirement vector for an ideal content representation based on the task. The ideal representation indicates the preferred content representation based on the task performed during the communication session between the users of user systems 110A-B. For example, when the task to be performed is a demonstration of an assembly of a machine an ideal requirement vector can include a high value associated with the body parameter. The ideal requirement is associated and adapted based on the task.

[0059] ASEM 120A then determines, based on the ideal requirement for a task, context, focus, and device, an effective requirement for displaying remote content. The effective requirement indicates changes made to the ideal requirement for a task based on context, focus, and device. The effective requirement vector can be expressed as: Reffective= (Fideal* Fcontext* Ffocus* Fdevice) + (Bideal* Bcontext* Bfocus* Bdevice) + (Eideal * Econtext * Efocus * Edevice) …. (1)

[0060] In a parallel manner, ASEM 120B determines an ideal requirement vector for an ideal content representation based on the task. Further, ASEM 120B determines, based on the ideal requirement for a task, context, focus, and device, an effective requirement for displaying remote content. The effective requirement indicates changes made to the ideal requirement for a task based on context, focus, and device. The effective requirement vector can be determined according to equation (1).

[0061] In addition, ASEM 120A determines the maximum capability of streaming volumetric content across the three components F, B, and E. The capability depends on the range and extent of data capture in the user system 110A, as well as the network parameters of both the uplink connection of the user system 110A, and the user system 110B downlink connection. Regarding extent and range of remote capture: the value of each component of the capability vector varies from 1 to 0 based on the capability of a system to capture a range of content representations varying between full volumetric representation to no spatialrepresentation. Regarding network parameters (remote and local): the streaming capabilities of a user system are directly related to the uplink and downlink bandwidth, network packet loss, jitter, and latency. ASEM 120A monitors these values and correspondingly scales the capability vector to represent the maximum possible fidelity of representations which can be reliably streamed at a high Quality of Service. Thus, the net capability vector can be represented as: Ceffective= (Fcapture* Fnetwork) + (Bcapture* Bnetwork) + (Ecapture* Enetwork) … (2)

[0062] In a parallel manner, ASEM 120B determines the maximum capability of streaming volumetric content across the three components F, B, and E. In some embodiments, the capability vector can be determined according to equation (2).

[0063] As an example, the following table represents the types of capture possible, the corresponding representations, and the approximate bandwidth cost. Capture Capabilities Content Representation Bandwidth Costdwidth.

[0064] Figure 4 illustrates exemplary representations with different content modalities.401 illustrates an exemplary facial avatar (low bandwidth) content representation modality.402 illustrates an exemplary spatial video (medium bandwidth) content representation modality.403 illustrates an exemplary point cloud (high bandwidth) content representation modality.404 illustrates an exemplary AR modulation content representation modality.

[0065] Figure 5 illustrates an exemplary flow of operations that can be performed in an ASEM for modulating content representation for a user system, in accordance with some embodiments. The operations described herein can be performed by one or more of ASEM 120A-B.

[0066] At operation 501, ASEM 120A determines a requirement parameter for user system 110A. In some embodiments, the requirement parameter is an effective requirementvector for user system 110A, indicating one or more requirements associated with multiple communication parameters. In some embodiments, ASEM 120A further determines the capability vector for user system 110A. At operation 502, ASEM 120A transmits an effective capability vector to ASEM 120B. In some embodiments, ASEM 120A also transmits an effective requirement vector to ASEM 120B. At operation 503, ASEM 120A receives or accepts a capability parameter from ASEM 120B. In some embodiments, the capability parameter is an effective capability vector. In some embodiments, ASEM 120A may also receive an effective requirement vector from ASEM 120B. The ASEM 120A and ASEM 120B constantly exchange information about their requirements and capabilities. The exemplary operations herein will be described with respect to ASEM 120A. However, similar operations can be performed by ASEM 120B in parallel. At a given time t1, user system 110B has an effective capability C2, and user system 110A has an effective requirement R1.

[0067] At operation 504, ASEM 120A compares the capability parameter with the requirement parameter. For example, ASEM 120A compares the effective capability C2 of user system 110B with the effective requirement R1 of user system 110A. Based on the result of the comparison, ASEM 120A takes one or more actions to adapt content representation of media content for user system 110A and / or user system 110B. Additionally, based on a result of the comparison of the requirement and the capability parameter, the ASEM 120A determines whether to adapt one or more content representations for one or more of the first user system and the second user system.

[0068] In one embodiment, based on the difference between capabilities C and requirements R, an ASEM computes a negotiated vector [F, B, E], and cross-references this vector against a lookup table of representations ranked according to fidelity and effective Quality of Experience. The lookup table can be generated from user experiments where various modalities are rated based on their quality of experience for the same task at different levels of detail. The ASEM determines the target representation modality for each of F, B, and E. The lookup table can further include information about “linked modalities” which offer similar levels of QoE while being significantly different in terms of fidelity. For example, for the face parameter, a mesh modality can be considered equivalent in terms of QoE as a video or a 3D avatar modality. In another example, for the body parameter, a mesh modality can be considered equivalent in terms of QoE as a 3D avatar modality. In a third example, for the environment parameter, a mesh modality can be considered equivalent to the video modality. As will be described below, these linked representations can be used by the ASEM to either seamlessly transition between modalities while maintaining acceptable QoE or offer users the option to dynamically choose between linked representations.

[0069] An example of the modalities in such a lookup table is as follows (linked representations in bold): Parameter Value Face Body Environment 0.9 - 1.0 Mesh Mesh Mesh

[0070] In one example, when considering face-to-face communication in 3D where users are moving around each other, an animated avatar with clearly articulated expressions offers a comparable quality of experience to a 3D Depth Image and Point Cloud. As mentioned in Table 1, the bandwidth cost of the avatar is much lesser than the point cloud, and thus in situations where streaming 3D information at a high quality is not feasible, the ASEMs can negotiate and switch to streaming media content that supports the highest quality of the animated avatar, as opposed to a poor-quality point cloud.

[0071] The comparison of the effective capability C2 and the effective requirement R1 can result in one of the following scenarios: the effective capability C2 of user system 110B is greater than the effective requirement R1 of user system 110A; the effective requirement R1 of user system 110A and the difference between the two is greater than the predetermined threshold; or C2 and R1 are comparable and fluctuating.

[0072] Responsive to determining that the effective capability C2 of user system 110B is greater than the effective requirement R1 of user system 110A and the difference between the two is greater than a predetermined threshold (C2 > R1 & |C2 – R1| > Threshold), ASEM 120A can perform the following operations: ASEM 120A negotiates with ASEM 120B the streaming and receiving of media content (e.g., spatial information) at a content representation level much higher than what is required at user system 110A. Following the negotiation, user system 110B streams media content at a level of capability C2. At operation 506, ASEM 120A receives from user system 110B the media content that is streamed at the capability level of C2. Alternatively, ASEM 120A receives the media content that is streamed at a negotiation capability level that is lower or equal to capability C2. Upon receipt of the media content, ASEM 120A determines, at operation 507, how much of that content is to be rendered. In some embodiments, the determination of how much of the content is to be rendered is based at least in part on therequirement for the first user system. In some embodiments, the threshold can be determined based on the reliability of the network, and the buffer required for smooth communication between user systems 110A-B.

[0073] Responsive to determining that the effective capability C2 of user system 110B is less than the effective requirement R1 of user system 110A and the difference between the two is greater than the predetermined threshold (C2 < R1 & |C2 – R1| > Threshold), ASEM 120A can perform the following operations: ASEM 120A receives from ASEM 120B the media content at the capability C2 of user system 110B, operation 508. Upon receipt of the media content, ASEM 120A determines, based on a task of the remote communication between the first user and the second user, at least one of a modality and a quality level for the media content that provides a similar Quality of Experience than a Quality of Experience obtained when the second user system has a capability parameter that satisfies the requirement parameter of the first user system. For example, ASEM 120A determines modalities and qualities that offer a functionally similar Quality of Experience based on a lookup table associated with the task to be performed. Table 2 represents a non-limiting example of such look up table.

[0074] Responsive to determining that C2 and R1 are comparable and fluctuating, |C2 – R1| < Threshold, the ASEMs cannot reliably provide the highest possible content representation based on the requirements. In such a situation, the ASEMs can negotiate in such a manner where ASEM 120B streams content at its highest capability, and ASEM 120A modulates the content representation of user system 110B based on the magnitude of difference between C2 and R1 by either modifying the quality of a certain representation or moving to different representations. Further, ASEM 120A can take into account predicted capability values to minimize the number of representation switches.

[0075] In a non-limiting example, the user (U1) of user system 110A demonstrates the assembly of a machine to user (U2) of user system 110B. In this example, initially, U2 wants to focus more on the process of assembly – U1’s hand gestures, and the alignment of parts – thus, the ideal requirement vector can be specified as (0.2, 0.8, 1.0). U2 constantly remains at a close distance to the shared 3D representation, and orbits the representation in a fixed angular window, meaning the “Context” and “Focus” multiplier remain high. But because U2 is viewing this information on a mobile device, the “Device” multiplier is low as even lower fidelity representations can be presented at high quality of experience on 2D displays. Thus, the effective requirement vector for user system 110B can be reduced to a net value of (0.1, 0.4, 0.5).

[0076] In one example, user system 110A includes a full depth capture suite, with associated processing power. This indicates that the ideal capability vector is consistently high(0.9, 0.9, 0.9). The network bandwidth at both locations is more than sufficient and highly reliable, therefore the network multiplier is also close to 1.0 on all parameters. Thus, assuming an effective capability vector of (0.8, 0.8, 0.8), the capability of streaming at the user system 110A greatly exceeds the requirements for display, where the effective requirement vector for user system 110B is (0.1, 0.4, 0.5).

[0077] ASEM 120A then streams media content at maximum capability. ASEM 120B accepts this information and begins by rendering the representation corresponding to the requirement vector from the lookup table associated with the type of tasks that include demonstration of machine assembly (e.g., the lookup table associated with object-oriented tasks), which in this case is a point-cloud.

[0078] As U2 moves closer to the object, thereby increasing the focus multiplier and thus the requirement vector, ASEM 120B already has access to a higher fidelity representation (from the received content, since ASEM 120A is streaming at maximum capability), and can seamlessly improve the level of detail or change modalities (from point cloud to mesh).

[0079] Suddenly, the uplink bandwidth for user system 110A drastically reduces, thereby reducing its effective capability vector to a low (0.3, 0.3, 0.3), which is what ASEM 120A starts streaming at. This is not sufficient to meet the effective requirement vector for user system 110B. Thus, upon receipt of the updated capability vector for user system 110A, ASEM 120B performs a lookup within acceptable thresholds of Quality of Experience variation and determines that it is best to transition to the linked representation of a video feed for the environment, and a 3D avatar of hand gestures for the body. While in this example, ASEM 120B may determine to transition the content representation for user system 110B, in other examples, ASEM 120B may determine to keep the existing content representation and not transition, even if the capability vector has decreased. The determination to not transition the content representation can be based on a predicted capability vector received from ASEM 120A that indicates that the network resources will be back up to their original capacity in a short period of time. Thus, ASEM 120B can absorb a potential reduction / degradation of QoE for U2 that will be limited in time.

[0080] In a few minutes, the original network capability is restored. ASEM 120A returns to streaming at a high capability, and ASEM 120B prompts the user (e.g., via a user interface) that the system can either automatically return to a higher-fidelity representation, or manually transition between the various higher linked representations available based on user input.

[0081] For a particularly detailed assembly maneuver, U2 decides they need to view the demonstration in VR. They indicate this to the ASEM 120B, exit the meeting from their mobile device, and join the same room via a VR headset by using the same call ID. ASEM 120B re-initializes on the headset, and carries forward the same capability and requirement vectors but modifies them based on the device. Now the requirement vector greatly increases due to the increased resolution and immersion of the VR display, and thus ASEM 120B moves to a higher- fidelity representation – that of a mesh – using the information that ASEM 120A has been streaming (which was already at a high level due to a higher capability vector).

[0082] Thus, the dynamic modulation of representations enabled by constant communication between the two ASEMs helps enable seamless remote communication and adaptation of content representation modalities across network conditions, task contexts, and device types and capabilities.

[0083] A similar approach is applied to content being streamed from user system 110B to user system 110A. Further, the same approach can be extended to multi-user remote calls, where each of the various ASEMs are in constant communication with each other and can selectively negotiate along specific channels to maintain different content representations across users.

[0084] This scenario is one example of many possible configurations that ASEMs and dynamic content representations enable. Other possible scenarios for modulation / adaptation include: prioritizing facial communication in degraded network contexts by transitioning to 3D avatars instead of lower quality video / point clouds, as QoE is maintained when expression fidelity is high; modulating the quality of 3D representations by switching between depth images and point cloud based on the focus requirements – when U1 views U2 head-on, depth is difficult to perceive, and a low-resolution depth image might suffice, but when U2 turns around to point to an object on the left, the representation will need to change to a higher resolution point cloud (in the case of high capability networks) or a video feed (in case of low capability networks) to minimize uncanniness.

[0085] Figure 6 shows an example of a communication system 600 in accordance with some embodiments.

[0086] In the example, the communication system 600 includes a cellular telecommunication network 602 that includes an access network 604, such as a radio access network (RAN), and a core network 606, which includes one or more core network nodes 608. The access network 604 includes one or more access network nodes, such as network nodes 610a and 610b (one or more of which may be generally referred to as network nodes 610), or any other similar 3rdGeneration Partnership Project (3GPP) access nodes or non-3GPP access points. Moreover, as will be appreciated by those of skill in the art, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments,the telecommunication network 602 includes one or more Open-RAN (ORAN) network nodes. An ORAN network node is a node in the telecommunication network 602 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other nodes to implement one or more functionalities of any node in the telecommunication network 602, including one or more network nodes 610 and / or core network nodes 608.

[0087] Examples of an ORAN network node include an open radio unit (O-RU), an open distributed unit (O-DU), an open central unit (O-CU), including an O-CU control plane (O-CU- CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time control application (e.g., xApp) or a non-real time control application (e.g., rApp), or any combination thereof (the adjective “open” designating support of an ORAN specification). The network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an A1, F1, W1, E1, E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN access node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment (described further below) in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an O-2 interface defined by the O-RAN Alliance or comparable technologies. The network nodes 610 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs 612a, 612b, 612c, and 612d (one or more of which may be generally referred to as UEs 612) to the core network 606 over one or more wireless connections. The UEs 612 may be the user systems 110A and / or 110B. Other aspects of the user systems 110 may be described herein with respect to UE 700 of Figure 7 and UE 1106 of Figure 11.

[0088] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 600 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 600 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0089] The UEs 612 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes 610 and other communication devices. Similarly, the network nodes 610 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs 612 and / or with other network nodes or equipment in the telecommunication network 602 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network 602.

[0090] In the depicted example, the core network 606 connects the network nodes 610 to one or more hosts, such as host 616. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 606 includes one more core network nodes (e.g., core network node 608) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 608. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0091] The host 616 may be under the ownership or control of a service provider other than an operator or provider of the access network 604 and / or the telecommunication network 602, and may be operated by the service provider or on behalf of the service provider. The host 616 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0092] As a whole, the communication system 600 of Figure 6 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or othersuitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low- power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0093] In some examples, the telecommunication network 602 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 602 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 602. For example, the telecommunications network 602 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC) / Massive IoT services to yet further UEs.

[0094] In some examples, the UEs 612 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 604 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 604. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved- UMTS Terrestrial Radio Access Network) New Radio – Dual Connectivity (EN-DC).

[0095] In the example, the hub 614 communicates with the access network 604 to facilitate indirect communication between one or more UEs (e.g., UE 612c and / or 612d) and network nodes (e.g., network node 610b). In some examples, the hub 614 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 614 may be a broadband router enabling access to the core network 606 for the UEs. As another example, the hub 614 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 610, or by executable code, script, process, or other instructions in the hub 614. As another example, the hub 614 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 614 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, the hub 614 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 614 then provides to the UE either directly, after performing localprocessing, and / or after adding additional local content. In still another example, the hub 614 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy IoT devices.

[0096] The hub 614 may have a constant / persistent or intermittent connection to the network node 610b. The hub 614 may also allow for a different communication scheme and / or schedule between the hub 614 and UEs (e.g., UE 612c and / or 612d), and between the hub 614 and the core network 606. In other examples, the hub 614 is connected to the core network 606 and / or one or more UEs via a wired connection. Moreover, the hub 614 may be configured to connect to an M2M service provider over the access network 604 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 610 while still connected via the hub 614 via a wired or wireless connection. In some embodiments, the hub 614 may be a dedicated hub – that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 610b. In other embodiments, the hub 614 may be a non-dedicated hub – that is, a device which is capable of operating to route communications between the UEs and network node 610b, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0097] Figure 7 shows a UE 700 in accordance with some embodiments. As used herein, a UE refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle, vehicle-mounted or vehicle embedded / integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0098] A UE may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle- to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is not intended for sale to, or operation by, anend user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0099] The UE 700 includes processing circuitry 702 that is operatively coupled via a bus 704 to an input / output interface 706, a power source 708, a memory 710, a communication interface 712, and / or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in Figure 7. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0100] The processing circuitry 702 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 710. The processing circuitry 702 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 702 may include multiple central processing units (CPUs).

[0101] In the example, the input / output interface 706 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE 700. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0102] In some embodiments, the power source 708 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 708 may further includepower circuitry for delivering power from the power source 708 itself, and / or an external power source, to the various parts of the UE 700 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 708. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 708 to make the power suitable for the respective components of the UE 700 to which power is supplied.

[0103] The memory 710 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read- only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 710 includes one or more application programs 714, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 716. The memory 710 may store, for use by the UE 700, any of a variety of various operating systems or combinations of operating systems.

[0104] The memory 710 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 710 may allow the UE 700 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 710, which may be or comprise a device-readable storage medium.

[0105] The processing circuitry 702 may be configured to communicate with an access network or other network using the communication interface 712. The communication interface 712 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 722. The communication interface 712 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a networknode in an access network). Each transceiver may include a transmitter 718 and / or a receiver 720 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 718 and receiver 720 may be coupled to one or more antennas (e.g., antenna 722) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0106] In the illustrated embodiment, communication functions of the communication interface 712 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short- range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / internet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0107] Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface 712, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0108] As another example, a UE comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0109] A UE, when in the form of an Internet of Things (IoT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an IoT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voicecontrolled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a head-mounted display for Augmented Reality (AR) or Virtual Reality (VR), a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an IoT device comprises circuitry and / or software in dependence of the intended application of the IoT device in addition to other components as described in relation to the UE 700 shown in Figure 7.

[0110] As yet another specific example, in an IoT scenario, a UE may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another UE and / or a network node. The UE may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-IoT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0111] In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

[0112] Figure 8 shows a network node 800 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a UE and / or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)), O-RAN nodes or components of an O-RAN node (e.g., O-RU, O-DU, O-CU).

[0113] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units, distributed units (e.g., in an O-RAN access node) and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0114] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0115] The network node 800 includes a processing circuitry 802, a memory 804, a communication interface 806, and a power source 808. The network node 800 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node 800 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 800 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 804 for different RATs) and some components may be reused (e.g., a same antenna 810 may be shared by different RATs). The network node 800 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 800, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 800.

[0116] The processing circuitry 802 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node 800 components, such as the memory 804, to provide network node 800 functionality.

[0117] In some embodiments, the processing circuitry 802 includes a system on a chip (SOC). In some embodiments, the processing circuitry 802 includes one or more of radio frequency (RF) transceiver circuitry 812 and baseband processing circuitry 814. In some embodiments, the radio frequency (RF) transceiver circuitry 812 and the baseband processing circuitry 814 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 812 and baseband processing circuitry 814 may be on the same chip or set of chips, boards, or units.

[0118] The memory 804 may comprise any form of volatile or non-volatile computer- readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry 802. The memory 804 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 802 and utilized by the network node 800. The memory 804 may be used to store any calculations made by the processing circuitry 802 and / or any data received via the communication interface 806. In some embodiments, the processing circuitry 802 and memory 804 is integrated.

[0119] The communication interface 806 is used in wired or wireless communication of signaling and / or data between a network node, access network, and / or UE. As illustrated, the communication interface 806 comprises port(s) / terminal(s) 816 to send and receive data, for example to and from a network over a wired connection. The communication interface 806 also includes radio front-end circuitry 818 that may be coupled to, or in certain embodiments a part of, the antenna 810. Radio front-end circuitry 818 comprises filters 820 and amplifiers 822. The radio front-end circuitry 818 may be connected to an antenna 810 and processing circuitry 802. The radio front-end circuitry may be configured to condition signals communicated between antenna 810 and processing circuitry 802. The radio front-end circuitry 818 may receive digitaldata that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 818 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 820 and / or amplifiers 822. The radio signal may then be transmitted via the antenna 810. Similarly, when receiving data, the antenna 810 may collect radio signals which are then converted into digital data by the radio front-end circuitry 818. The digital data may be passed to the processing circuitry 802. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0120] In certain alternative embodiments, the network node 800 does not include separate radio front-end circuitry 818, instead, the processing circuitry 802 includes radio front-end circuitry and is connected to the antenna 810. Similarly, in some embodiments, all or some of the RF transceiver circuitry 812 is part of the communication interface 806. In still other embodiments, the communication interface 806 includes one or more ports or terminals 816, the radio front-end circuitry 818, and the RF transceiver circuitry 812, as part of a radio unit (not shown), and the communication interface 806 communicates with the baseband processing circuitry 814, which is part of a digital unit (not shown).

[0121] The antenna 810 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna 810 may be coupled to the radio front-end circuitry 818 and may be any type of antenna capable of transmitting and receiving data and / or signals wirelessly. In certain embodiments, the antenna 810 is separate from the network node 800 and connectable to the network node 800 through an interface or port.

[0122] The antenna 810, communication interface 806, and / or the processing circuitry 802 may be configured to perform any receiving operations and / or certain obtaining operations described herein as being performed by the network node. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, the antenna 810, the communication interface 806, and / or the processing circuitry 802 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and / or signals may be transmitted to a UE, another network node and / or any other network equipment.

[0123] The power source 808 provides power to the various components of network node 800 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 808 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 800 with power for performing the functionality described herein. For example, the network node 800 may be connectable to an external power source (e.g., the power grid, an electricity outlet) viaan input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 808. As a further example, the power source 808 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0124] Embodiments of the network node 800 may include additional components beyond those shown in Figure 8 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the network node 800 may include user interface equipment to allow input of information into the network node 800 and to allow output of information from the network node 800. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 800.

[0125] Figure 9 is a block diagram of a host 900, which may be an embodiment of the host 616 of Figure 6, in accordance with various aspects described herein. As used herein, the host 900 may be or comprise various combinations hardware and / or software, including a standalone server, a blade server, a cloud-implemented server, a distributed server, a virtual machine, container, or processing resources in a server farm. The host 900 may provide one or more services to one or more UEs. The host 900 may provide an embodiment of an ASEM, such as ASEM 120A and / or ASEM 120B, as described herein.

[0126] The host 900 includes processing circuitry 902 that is operatively coupled via a bus 904 to an input / output interface 906, a network interface 908, a power source 910, and a memory 912. Other components may be included in other embodiments. Features of these components may be substantially similar to those described with respect to the devices of previous figures, such as Figures 7 and 8, such that the descriptions thereof are generally applicable to the corresponding components of host 900.

[0127] The memory 912 may include one or more computer programs including one or more host application programs 914 and data 916, which may include user data, e.g., data generated by a UE for the host 900 or data generated by the host 900 for a UE. Embodiments of the host 900 may utilize only a subset or all of the components shown. The host application programs 914 may be implemented in a container-based architecture and may provide support for video codecs (e.g., Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), MPEG, VP9) and audio codecs (e.g., FLAC, Advanced Audio Coding (AAC), MPEG, G.711), including transcoding for multiple different classes, types, or implementations of UEs (e.g., handsets, desktop computers, wearable display systems, heads-up display systems). The host application programs 914 may also provide for user authenticationand licensing checks and may periodically report health, routes, and content availability to a central node, such as a device in or on the edge of a core network. Accordingly, the host 900 may select and / or indicate a different host for over-the-top services for a UE. The host application programs 914 may support various protocols, such as the HTTP Live Streaming (HLS) protocol, Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), Dynamic Adaptive Streaming over HTTP (MPEG-DASH), etc.

[0128] Figure 10 is a block diagram illustrating a virtualization environment 1000 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 1000 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 1000 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an O-2 interface.

[0129] Applications 1002 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0130] Hardware 1004 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1006 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 1008a and 1008b (one or more of which may be generally referred to as VMs 1008), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 1006 may present a virtual operating platform that appears like networking hardware to the VMs 1008.

[0131] The VMs 1008 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 1006. Different embodiments of the instance of a virtual appliance 1002 may be implemented on one or more of VMs 1008, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0132] In the context of NFV, a VM 1008 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 1008, and that part of hardware 1004 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 1008 on top of the hardware 1004 and corresponds to the application 1002.

[0133] Hardware 1004 may be implemented in a standalone network node with generic or specific components. Hardware 1004 may implement some functions via virtualization. Alternatively, hardware 1004 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 1010, which, among others, oversees lifecycle management of applications 1002. In some embodiments, hardware 1004 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1012 which may alternatively be used for communication between hardware nodes and radio units.

[0134] Figure 11 shows a communication diagram of a host 1102 communicating via a network node 1104 with a UE 1106 over a partially wireless connection in accordance with some embodiments. Example implementations, in accordance with various embodiments, of the UE (such as a UE 612a of Figure 6 and / or UE 700 of Figure 7), network node (such as network node 610a of Figure 6 and / or network node 800 of Figure 8), and host (such as host 616 of Figure 6 and / or host 900 of Figure 9) discussed in the preceding paragraphs will now be described with reference to Figure 11.

[0135] Like host 900, embodiments of host 1102 include hardware, such as a communication interface, processing circuitry, and memory. The host 1102 also includes software, which is stored in or accessible by the host 1102 and executable by the processing circuitry. The software includes a host application that may be operable to provide a service to a remote user, such as the UE 1106 connecting via an over-the-top (OTT) connection 1150 extending between the UE 1106 and host 1102. In providing the service to the remote user, a host application may provide user data which is transmitted using the OTT connection 1150.

[0136] The network node 1104 includes hardware enabling it to communicate with the host 1102 and UE 1106. The connection 1160 may be direct or pass through a core network (like core network 606 of Figure 6) and / or one or more other intermediate networks, such as one or more public, private, or hosted networks. For example, an intermediate network may be a backbone network or the Internet.

[0137] The UE 1106 includes hardware and software, which is stored in or accessible by UE 1106 and executable by the UE’s processing circuitry. The software includes a client application, such as a web browser or operator-specific “app” that may be operable to provide a service to a human or non-human user via UE 1106 with the support of the host 1102. In the host 1102, an executing host application may communicate with the executing client application via the OTT connection 1150 terminating at the UE 1106 and host 1102. In providing the service to the user, the UE's client application may receive request data from the host's host application and provide user data in response to the request data. The OTT connection 1150 may transfer both the request data and the user data. The UE's client application may interact with the user to generate the user data that it provides to the host application through the OTT connection 1150.

[0138] The OTT connection 1150 may extend via a connection 1160 between the host 1102 and the network node 1104 and via a wireless connection 1170 between the network node 1104 and the UE 1106 to provide the connection between the host 1102 and the UE 1106. The connection 1160 and wireless connection 1170, over which the OTT connection 1150 may be provided, have been drawn abstractly to illustrate the communication between the host 1102 and the UE 1106 via the network node 1104, without explicit reference to any intermediary devices and the precise routing of messages via these devices.

[0139] As an example of transmitting data via the OTT connection 1150, in step 1108, the host 1102 provides user data, which may be performed by executing a host application. In some embodiments, the user data is associated with a particular human user interacting with the UE 1106. In other embodiments, the user data is associated with a UE 1106 that shares data with the host 1102 without explicit human interaction. In step 1110, the host 1102 initiates a transmission carrying the user data towards the UE 1106. The host 1102 may initiate thetransmission responsive to a request transmitted by the UE 1106. The request may be caused by human interaction with the UE 1106 or by operation of the client application executing on the UE 1106. The transmission may pass via the network node 1104, in accordance with the teachings of the embodiments described throughout this disclosure. Accordingly, in step 1112, the network node 1104 transmits to the UE 1106 the user data that was carried in the transmission that the host 1102 initiated, in accordance with the teachings of the embodiments described throughout this disclosure. In step 1114, the UE 1106 receives the user data carried in the transmission, which may be performed by a client application executed on the UE 1106 associated with the host application executed by the host 1102.

[0140] In some examples, the UE 1106 executes a client application which provides user data to the host 1102. The user data may be provided in reaction or response to the data received from the host 1102. Accordingly, in step 1116, the UE 1106 may provide user data, which may be performed by executing the client application. In providing the user data, the client application may further consider user input received from the user via an input / output interface of the UE 1106. Regardless of the specific manner in which the user data was provided, the UE 1106 initiates, in step 1118, transmission of the user data towards the host 1102 via the network node 1104. In step 1120, in accordance with the teachings of the embodiments described throughout this disclosure, the network node 1104 receives user data from the UE 1106 and initiates transmission of the received user data towards the host 1102. In step 1122, the host 1102 receives the user data carried in the transmission initiated by the UE 1106.

[0141] One or more of the various embodiments improve the performance of OTT services provided to the UE 1106 using the OTT connection 1150, in which the wireless connection 1170 forms the last segment.

[0142] In an example scenario, factory status information may be collected and analyzed by the host 1102. As another example, the host 1102 may process audio and video data which may have been retrieved from a UE for use in creating maps. As another example, the host 1102 may collect and analyze real-time data to assist in controlling vehicle congestion (e.g., controlling traffic lights). As another example, the host 1102 may store surveillance video uploaded by a UE. As another example, the host 1102 may store or control access to media content such as video, audio, VR or AR which it can broadcast, multicast or unicast to UEs. As other examples, the host 1102 may be used for energy pricing, remote control of non-time critical electrical load to balance power generation needs, location services, presentation services (such as compiling diagrams etc. from data collected from remote devices), or any other function of collecting, retrieving, storing, analyzing and / or transmitting data.

[0143] In some examples, a measurement procedure may be provided for the purpose of monitoring data rate, latency and other factors on which the one or more embodiments improve. There may further be an optional network functionality for reconfiguring the OTT connection 1150 between the host 1102 and UE 1106, in response to variations in the measurement results. The measurement procedure and / or the network functionality for reconfiguring the OTT connection may be implemented in software and hardware of the host 1102 and / or UE 1106. In some embodiments, sensors (not shown) may be deployed in or in association with other devices through which the OTT connection 1150 passes; the sensors may participate in the measurement procedure by supplying values of the monitored quantities exemplified above, or supplying values of other physical quantities from which software may compute or estimate the monitored quantities. The reconfiguring of the OTT connection 1150 may include message format, retransmission settings, preferred routing etc.; the reconfiguring need not directly alter the operation of the network node 1104. Such procedures and functionalities may be known and practiced in the art. In certain embodiments, measurements may involve proprietary UE signaling that facilitates measurements of throughput, propagation times, latency and the like, by the host 1102. The measurements may be implemented in that software causes messages to be transmitted, in particular empty or ‘dummy’ messages, using the OTT connection 1150 while monitoring propagation times, errors, etc.

[0144] Although the computing devices described herein (e.g., UEs, network nodes, hosts) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functionsof any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0145] The following clauses describe some enumerated embodiments of the disclosure.

[0146] Group A Embodiments

[0147] Embodiment 1. A method performed by a network node for adaptive spatial remote communication, the method comprising: determining a requirement parameter for a first user system; receiving a capability parameter for a second user system that is used by a second user in remote communication with a first user of the first user system; comparing the requirement parameter and the capability parameter; based on a result of the comparison of the requirement and the capability parameter, determining whether to adapt one or more content representations for one or more of the first user system and the second user system; and in response to determining that at least one of the content representations is to be adapted, causing at least one of the first user system and the second user system to adapt the at least one of the content representations to be presented to at least one of the first user and the second user, wherein the adapting is based on a first modality and / or a first quality level.

[0148] Embodiment 2. The method of the previous embodiment, wherein the determining whether to adapt one or more content representations of one or more of the first user system and the second user system includes: determining that the capability parameter for the second user system is greater than the requirement parameter for the first user system.

[0149] Embodiment 3. The method of the previous embodiment, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes: negotiating, based at least in part on the capability parameter for the second user system, a content representation for streaming and receiving media content from the second user system.

[0150] Embodiment 4. The method of the previous embodiment, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes: receiving, from the second user system, media content at the negotiated content representation.

[0151] Embodiment 5. The method of the previous embodiments, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes: determining, based at least in part on the requirement parameter for the first user system, how much of the media content is to be rendered.

[0152] Embodiment 6. The method of the first embodiment, wherein the determining whether to adapt one or more content representations of one or more of the first user system andthe second user system includes: determining the capability parameter of the second user system is less than the requirement parameter of the first user system.

[0153] Embodiment 7. The method of the previous embodiment, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes: receiving, from the second user system, media content at a content representation that is associated with the capability parameter for the second user system; and determining, based on a task of the remote communication between the first user and the second user, at least one of a modality and a quality level for the media content that provides a similar Quality of Experience than a Quality of Experience obtained when the second user system has a capability parameter that satisfies the requirement parameter of the first user system.

[0154] Embodiment 8. The method of the first embodiment, wherein the determining whether to adapt one or more content representations of one or more of the first user system and the second user system includes: determining that the capability parameter of the second user system and the requirement parameter of the first user system are comparable and fluctuating.

[0155] Embodiment 9. The method of the previous embodiment, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes negotiating, based at least in part on the capability parameter for the second user system, a content representation for streaming and receiving media content from the second user system, wherein the content representation is associated with a highest capability parameter for the second user system.

[0156] Embodiment 10. The method of any of the previous embodiments further including: receiving an updated capability parameter for the second user system; comparing the requirement parameter with the updated capability parameter; based on a result of the comparison of the requirement and the updated capability parameter, determining whether to adapt the content representations of one or more of the first user system and the second user system; and in response to determining that at least one of the content representations is to be adapted, causing at least one of the first user system and the second user system to adapt the at least one of the content representations to be presented to at least one of the first user and the second user, wherein the adapting is performed based on a second modality and / or second quality level that is different from the first modality and / or first level quality.

[0157] Group B Embodiments

[0158] Embodiment 11. A network node for adaptive spatial remote communication, the network node comprising: processing circuitry configured to perform any of the steps of any ofthe Group A embodiments; and power supply circuitry configured to supply power to the processing circuitry.

[0159] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer- readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

Claims

CLAIMS 1. A method performed by a host (900) for adaptive spatial remote communication, the method comprising: determining (501) a requirement parameter for a first user system; receiving (503) a capability parameter for a second user system that is used by a second user in remote communication with a first user of the first user system; comparing (504) the requirement parameter and the capability parameter; based on a result of the comparison of the requirement and the capability parameter, determining whether to adapt one or more content representations for one or more of the first user system and the second user system; and in response to determining that at least one of the content representations is to be adapted, causing (514) at least one of the first user system and the second user system to adapt the at least one of the content representations to be presented to at least one of the first user and the second user, wherein the adapting is based on a first modality and / or a first quality level.

2. The method of claim 1, wherein the determining whether to adapt one or more content representations of one or more of the first user system and the second user system includes: determining that the capability parameter for the second user system is greater than the requirement parameter for the first user system.

3. The method any of claims 1-2, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes: negotiating, based at least in part on the capability parameter for the second user system, a content representation for streaming and receiving media content from the second user system.

4. The method of claim 3, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes: receiving, from the second user system, media content at the negotiated content representation.

5. The method of claim 4, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes: determining, based at least in part on the requirement parameter for the first user system, how much of the media content is to be rendered.

6. The method of any of claims 1-5, wherein the determining whether to adapt one or more content representations of one or more of the first user system and the second user system includes: determining the capability parameter of the second user system is less than the requirement parameter of the first user system.

7. The method of claim 1, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes: receiving, from the second user system, media content at a content representation that is associated with the capability parameter for the second user system; and determining, based on a task of the remote communication between the first user and the second user, at least one of a modality and a quality level for the media content that provides a similar Quality of Experience than a Quality of Experience obtained when the second user system has a capability parameter that satisfies the requirement parameter of the first user system.

8. The method of claim 1, wherein the determining whether to adapt one or more content representations of one or more of the first user system and the second user system includes: determining that the capability parameter of the second user system and the requirement parameter of the first user system are comparable and fluctuating.

9. The method of claim 8, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes negotiating, based at least in part on the capability parameter for the second user system, a content representation for streaming and receiving media content from the second user system, wherein the content representation is associated with a highest capability parameter for the second user system.

10. The method of any of claims 1-9, further comprising: receiving an updated capability parameter for the second user system; comparing the requirement parameter with the updated capability parameter; based on a result of the comparison of the requirement and the updated capability parameter, determining whether to adapt the content representations of one or more of the first user system and the second user system; and in response to determining that at least one of the content representations is to be adapted, causing at least one of the first user system and the second user system to adapt the at least one of the content representations to be presented to at leastone of the first user and the second user, wherein the adapting is performed based on a second modality and / or second quality level that is different from the first modality and / or first level quality.

11. A host for adaptive spatial remote communication, the host comprising: processing circuitry configured to: determining a requirement parameter for a first user system; receive a capability parameter for a second user system that is used by a second user in remote communication with a first user of the first user system; compare the requirement parameter and the capability parameter; based on a result of the comparison of the requirement and the capability parameter, determine whether to adapt one or more content representations for one or more of the first user system and the second user system; and in response to determining that at least one of the content representations is to be adapted, cause at least one of the first user system and the second user system to adapt the at least one of the content representations to be presented to at least one of the first user and the second user, wherein the adapting is based on a first modality and / or a first quality level; and power supply circuitry configured to supply power to the processing circuitry.

12. The host of claim 11, wherein the determining whether to adapt one or more content representations of one or more of the first user system and the second user system includes: determining that the capability parameter for the second user system is greater than the requirement parameter for the first user system.

13. The host of any of claims 11-12, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes: negotiating, based at least in part on the capability parameter for the second user system, a content representation for streaming and receiving media content from the second user system.

14. The host of claim 13, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes: receiving, from the second user system, media content at the negotiated content representation.

15. The host of claim 14, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes: determining, based at least in part on the requirement parameter for the first user system, how much of the media content is to be rendered.

16. The host of any of claims 11-15, wherein the determining whether to adapt one or more content representations of one or more of the first user system and the second user system includes: determining the capability parameter of the second user system is less than the requirement parameter of the first user system.

17. The host of claim 11, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes: receiving, from the second user system, media content at a content representation that is associated with the capability parameter for the second user system; and determining, based on a task of the remote communication between the first user and the second user, at least one of a modality and a quality level for the media content that provides a similar Quality of Experience than a Quality of Experience obtained when the second user system has a capability parameter that satisfies the requirement parameter of the first user system.

18. The host of claim 11, wherein the determining whether to adapt one or more content representations of one or more of the first user system and the second user system includes: determining that the capability parameter of the second user system and the requirement parameter of the first user system are comparable and fluctuating.

19. The host of claim 11, wherein the causing at least one of the first user system and the second user system to adapt the at least one of the content representations includes negotiating, based at least in part on the capability parameter for the second user system, a content representation for streaming and receiving media content from the second user system, wherein the content representation is associated with a highest capability parameter for the second user system.

20. The host of any of claims 11-19, wherein the processing circuitry is further configured to: receive an updated capability parameter for the second user system; compare the requirement parameter with the updated capability parameter;based on a result of the comparison of the requirement and the updated capability parameter, determine whether to adapt the content representations of one or more of the first user system and the second user system; and in response to determining that at least one of the content representations is to be adapted, cause at least one of the first user system and the second user system to adapt the at least one of the content representations to be presented to at least one of the first user and the second user, wherein the adapting is performed based on a second modality and / or second quality level that is different from the first modality and / or first level quality.

Images