Methods, apparatus and systems for optimizing communication between sender(s) and receiver(s) in computer-mediated reality applications

Abstract

To provide systems, methods and apparatus for processing a media content for reproduction by a first apparatus.SOLUTION: A method includes obtaining pose information indicative of a position and / or orientation of a user. The pose information is transmitted to a second apparatus that provides a media content. The media content is rendered on the basis of the pose information to obtain the rendered media content. The rendered media content is transmitted to the first apparatus for reproduction. The first apparatus (receiver 550) obtains the pose information and transmits the pose information to the second apparatus. The second apparatus (transmitter 500) renders the media content on the basis of the pose information to obtain the rendered media content and transmits the rendered media content to the first apparatus for reproduction.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 680,678, filed June 5, 2018, as well as U.S. Provisional Patent Application No. 62 / 519,952, both filed June 15, 2017, and European Patent Application No. 17176248.7, all of which are incorporated by reference in their entirety.

[0002] Technical Field This disclosure relates to computer-mediated reality applications such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) applications, which may include, but are not limited to, client / receiver binaural and non-binaural audio and video applications. [Background technology]

[0003] Applications and products in computer-mediated reality spaces (e.g., VR, AR and MR spaces) are rapidly evolving to include increasingly sophisticated acoustic models of sound sources and scenes. Without intending to be limiting, VR, AR and MR are referred to in the remainder of this paper. To optimize the computer-mediated reality experience, it is preferable to minimize the delay between the user's movement (e.g., head movement) and the perception of sound adapted to this movement (rendered sound). This delay is also known as motion-to-sound latency or motion-to-ear lag. Furthermore, for a typical receiver device such as a smartphone, where optimizing computational complexity and power consumption is important, it is also desirable to minimize the number of instructions required to decode and render the sound. When the entire audio scene is transmitted, but not for example communication, the focus is on the rendering latency of the receiver. For example, linear applications (e.g., movies) do not react dynamically to the user's movements. However, for interactive content, all accumulated round-trip latencies would need to be considered (e.g., when a user triggers an event that needs to be sent back to the server for rendering). Dynamically changed content should be encoded with enough lead time before the content is consumed so that the user does not perceive any latency between the movement and the resulting effect, and so that there is no lag between the audio and video of the content. For linear applications, no encoding and decoding latency is considered for movement-to-audio latency, because user movements (position and / or orientation) do not affect the content itself; rather, these movements only affect the viewpoint from which the content is viewed. Thus, for linear content, user movements only affect the rendering, not the encoding and / or decoding of the output audio.For communications, it is different because the system can start encoding, transmitting, and decoding media as content (e.g. speech) occurs. The same is true if interactive content (e.g. from a game engine) is rendered and encoded in real-time by a remote server in the cloud. Furthermore, it is very important that the overall latency of the video and audio systems is the same, since differences can cause motion-sickness. Therefore, depending on the latency of the video system, it is necessary to achieve the same level of audio system latency. Summary of the Invention [Problem to be solved by the invention]

[0004] This document addresses the technical problem of typical AR, VR and MR systems, where too high latency and too high computational complexity are required to provide a convincing experience. To address this problem, this document proposes a method for processing media content, a system for processing media content and corresponding devices having the features of the respective independent claims. [Means for solving the problem]

[0005] An aspect of the present disclosure relates to a method of processing media content for reproduction by a first device. The first device may be, for example, one of a receiver, a receiver device, or a reproduction device. The first device may correspond to, include, or work in conjunction with an AR / VR / MR device, such as, for example, an AR / VR / MR headset. Thus, the first device may include reproduction equipment (e.g., speakers, headphones) for reproducing the media content, and a processor coupled to the reproduction equipment. The media content may be or include audio content and / or video content. The processing may involve or correspond to rendering. The reproduction may include or correspond to replay. The method may include obtaining pose information indicative of a position and / or orientation of a user. The obtaining of the pose information may be performed in the first device. The user may be a user of the first device. The pose information may relate to, for example, a head of the user. The pose information may be obtained by a sensor, e.g., a pose sensor, that may be positioned in alignment with the user. Thus, the pose information may be referred to as sensor data. The pose information may further include one or more first derivatives of the pose and / or one or more second derivatives of the pose. The user may, for example, be wearing an AR / VR / MR device. The method may further include transmitting the pose information to a second device that provides (e.g., stores, relays) the media content. The second device may, for example, be one of a sender device, a server device, or a content delivery device. The second device may be a device for providing the media content to the first device. The first and second devices may be spatially separated from each other. The method may further include rendering the media content based on the pose information to obtain a rendered media content. The rendering may be performed in the second device.The rendered media content may be referred to as pre-rendered media content. The rendering may be to two or more channels, e.g., in the case of audio content. The method may further include transmitting the rendered media content to a first device for reproduction. The transmitting of the rendered media content may be performed by a second device. The method may further include reproducing (e.g., playing) (by the first device) the rendered media content.

[0006] In the case where only the rendered media content needs to be transmitted, the transmission bitrate for lossless data transmission may be similar or equal to the bitrate of the compressed version of the complete media content. Thus, compression may not be necessary in the context of the proposed method. Transmitting an uncompressed or lossless media stream eliminates or reduces the latency for encoding and decoding. For example, the latency resulting from encoding / decoding may be reduced to zero, which also leads to an overall reduction in the motion-to-ear latency and / or motion-to-eye latency. Furthermore, in the absence of compression of the pre-rendered media content, the first device (receiver) may output the audio / video without decoding or rendering. This leads to a reduction in the computational complexity at the receiver, since no decoding needs to be performed and / or rendering has already been completed at the transmitting side. Thus, the proposed method may reduce the motion-to-ear latency and / or motion-to-eye latency, and may further reduce the computational complexity at the receiving side.

[0007] In some embodiments, the media content may include audio content and the rendered media content may include rendered audio content. Alternatively or additionally, the media content may include video content and the rendered media content may include rendered video content.

[0008] In some embodiments, the media content may include audio content and the rendered media content may include rendered audio content, where the method may further include generating an audible (e.g., sound) representation of the rendered audio content. The generation of the audible representation may be performed on the first device, for example, in the case of audio content, via two or more speakers of the first device.

[0009] In some embodiments, the audio content may be one of First Order Ambisonics (FOA)-based, Higher Order Ambisonics (HOA)-based, object-based, or channel-based audio content, or a combination of two or more of FOA-based, HOA-based, object-based, or channel-based audio content.

[0010] In some embodiments, the rendered audio content may be one of binaural audio content, FOA audio content, HOA audio content, or channel-based audio content, or a combination of two or more of binaural audio content, FOA audio content, HOA audio content, or channel-based audio content.

[0011] In some embodiments, the rendering may include obtaining predicted attitude information based on the attitude information and further based on previous attitude information and / or one or more first and / or second derivatives. The predicted attitude information may be attitude information for a future time point. The previous attitude information may be attitude information obtained at or received from the first device at a previous time point. The prediction may be performed at the second device. Alternatively, the prediction may be performed at the first device. In the latter case, the first device may transmit the predicted attitude information to the second device. The rendering may further include rendering the media content based on the predicted attitude information to obtain a rendered media content.

[0012] By taking into account the predicted pose information, delays that may result from encoding / decoding the rendered media content and / or transmitting the rendered media content to the first device can be addressed. In other words, said delays can be hidden due to the properly predicted pose information, so that the user may not notice this delay and may not perceive any mismatch between audio, video, and motion.

[0013] In some embodiments, the method may further include transmitting the predicted pose information to the first device along with the rendered media content.

[0014] This enables the first device to perform a check as to whether the predicted orientation information (i.e., in this case, the orientation information used to render the media content) is the same (or substantially the same) as the actual / current orientation information (i.e., the orientation information currently available on the first device) and to appropriately adapt the rendered media content if there is a discrepancy between the predicted orientation information and the actual / current orientation information.

[0015] In some embodiments, the method may further include comparing the predicted pose information with the actual pose information. The method may further include updating the rendered media content based on a result of the comparison. The comparison and the updating may be performed on the first device. The actual pose information may be pose information at (e.g., obtained at) a time when the rendered media content is played, e.g., by the first device. The updating may be performed, e.g., based on a difference between the predicted pose information and the actual pose information. The updating may involve extrapolation of the rendered media content, e.g., by rotation, level change, and / or blind upmix.

[0016] In some embodiments, the predicted pose information may be predicted for an estimate of a time when the rendered media content is expected to be processed by the first device for reproduction. The processing of the rendered media content by the first device may involve reproduction (e.g., playing) of the rendered media content. The actual pose information (e.g., current pose information) may be pose information obtained at a time when the rendered media content is actually processed by the first device for reproduction. The actual pose information may be obtained at a time when the rendered media content is actually processed by the first device.

[0017] Thereby, mismatches between predicted and actual pose information can be addressed, thereby allowing a better adaptation of the rendered media content to the user's pose (e.g., the user's head pose) and avoiding discrepancies between the user's perceived audio / video scene and the expected audio / video scene. Since the mismatch between predicted and actual pose information is expected to be small, such adaptation can be safely left to the first device with tractable computational complexity.

[0018] In some embodiments, the rendered media content may be transmitted to the first device in uncompressed form.

[0019] This allows to reduce the computational complexity in the first device (receiver) and further reduces the round trip delay between the change in pose and the reproduction of the media content rendered according to the changed pose.

[0020] In some embodiments, the method may further include encoding (e.g., compressing) the rendered media content prior to transmission to the first device. The method may also include decoding (e.g., decompressing) the encoded rendered media content after receipt at the first device. The encoding / decoding may include or correspond to compressing / decompressing the rendered media content. The encoding / decoding may be low-latency encoding / decoding.

[0021] In some embodiments, the estimate of the time at which the rendered audio content is expected to be processed by the first device for reproduction may include an estimate of the time required to encode and decode the rendered audio content and / or an estimate of the time required to transmit the rendered media content to the first device.

[0022] In some embodiments, the predicted pose information may be obtained further based on an estimate of a time required to encode and decode the rendered media content and / or an estimate of a time required to transmit the rendered media content to the first device.

[0023] Thereby, delays resulting from encoding / decoding and / or transmission can be hidden, in the sense that the user is not aware of these delays.

[0024] In some embodiments, the method may further include comparing pose information used to render the media content with current pose information. The current pose information may, for example, be pose information available at the time of reproducing the rendered media content. The method may also further include updating the rendered media content based on a result of the comparison. The updating may be performed based on a difference between the pose information used to render the media content and the current pose information. The updating may involve extrapolation of the rendered media content, for example, by rotation, level change and / or blind upmix.

[0025] In some embodiments, the method may further include determining, at the second device, gradient information indicative of how the rendered media content changes in response to changes in the pose information (e.g., a change in pose). The gradient information may be indicative of changes in subband energy levels (e.g., for each channel) in response to a translation and / or rotation of a user (e.g., a user's head) (for audio content). The method may further include transmitting the gradient information to the first device together with the rendered media content. The method may further include comparing, at the first device, pose information used to render the media content with current pose information. The pose information used (by the second device) to render the media content may be transmitted to the first device together with the rendered media content. If the pose information is not transmitted to the first device together with the rendered media content, the first device may refer to pose information it sent to the second device. The current pose information may, for example, be pose information available at the time of reproducing the rendered media content. The method may further include updating the rendered media content based on the gradient information and a result of the comparison. An update of the rendered media content may be performed based on a difference between pose information used to render the media content and the current pose information, said update may involve extrapolation of the rendered media content, for example due to rotation, level changes and / or blind upmixing.

[0026] Thereby, small imperfections in the prediction of pose information can be corrected and any mismatch between the pose and the reproduced media content can be avoided.

[0027] In some embodiments, the media content may include audio content and the rendered media content may include rendered audio content, where the method may further include transmitting environmental information to the second device indicative of acoustic characteristics of an environment in which the first device is located. In this case, the rendering of the media content may be further based on the environmental information. The environmental information may include room characteristics and / or Binaural Room Impulse Response (BRIR) functions.

[0028] This allows the reproduced media content to be specifically adapted to the particular environment in which the user is located, thereby enhancing the user's computer-mediated reality experience.

[0029] In some embodiments, the media content may include audio content and the rendered media content may include rendered audio content. In that case, the method may further include transmitting morphological information indicative of a morphology of the user or a part of the user to the second device. In this case, the rendering of the media content may be further based on the morphological information. The morphology may include or correspond to a shape or size, for example, a shape or size of the user's head. The morphological information may include Head-Related Transfer Functions (HRTFs). The rendering may be a binaural rendering.

[0030] This allows the reproduced media content to be specifically adapted to the particular morphology of a user or part of a user, thereby enhancing the user's computer-mediated reality experience.

[0031] Further aspects of the present disclosure relate to a first device, a second device and a system of the first device and the second device based on (eg, implementing) the above aspects and embodiments thereof.

[0032] Thus, another aspect of the present disclosure relates to a system having a first device for reproducing media content and a second device storing the media content. The first device may be adapted to obtain pose information indicative of a position and / or orientation of a user. The first device may be further adapted to transmit the pose information to the second device. The second device may be adapted to render the media content based on the pose information to obtain a rendered media content. The second device may be further adapted to transmit the rendered media content to the first device for reproduction. For example, the first and second devices may include a respective processor (or a respective set of processors) and a memory coupled to the respective processor (or the respective set of processors). The processors may be adapted to perform the above operations.

[0033] Another aspect of the present disclosure relates to a second device for providing media content for reproduction by a first device. The second device may be adapted to receive pose information indicative of a position and / or orientation of a user of the first device. The second device may be further adapted to render the media content based on the pose information to obtain a rendered media content. The second device may also be adapted to transmit the rendered media content to the first device for reproduction. For example, the second device may include a processor (or a set of processors) and a memory coupled to the processor (or set of processors). The processor (or set of processors) may be adapted to perform the above operations.

[0034] Another aspect of the present disclosure relates to a first device for reproducing media content provided by a second device. The first device may be adapted to obtain pose information indicative of a position and / or orientation of a user of the first device. The first device may be further adapted to transmit the pose information to the second device. The first device may be further adapted to receive rendered media content from the second device. The rendered media content may be obtained by rendering the media content based on the pose information. The first device may also be adapted to reproduce the rendered media content. For example, the first device may include a processor (or a set of processors) and a memory coupled to the processor (or set of processors). The processor (or set of processors) may be adapted to perform the above operations.

[0035] It should be noted that statements made with respect to methods also apply to the corresponding systems and apparatus used in such methods / systems, and vice versa.

[0036] Further aspects of the disclosure relate to systems, devices, methods, and computer-readable storage media configured to perform a method for rendering audio content, including receiving user position and / or orientation data by a transmitter (S) device and transmitting corresponding pre-rendered content, typically derived from an object-based or FOA / HOA representation. The pre-rendered signal generated by the transmitter can be binaural, FOA, HOA, or any type of channel-based rendering. The method may further include transmitting the uncompressed pre-rendered content. The method may further include encoding the pre-rendered content and transmitting the encoded pre-rendered content. The method may further include receiving the pre-rendered content by a receiver. The method may further include decoding the pre-rendered pre-encoded binauralized content by the receiver. The user position and / or orientation data may include a local pose indicating the user's position and orientation in world space. The user position data may be transmitted from the receiver to the transmitter. The method may further include sending back to the receiver the user position data used for the pre-rendered binauralized content. The method may further include extrapolating the pre-rendered content to determine updated content based on the received user position data and local position data. The method may further include transmitting morphological data (e.g., head size) about the user for individualized binaural processing. The method may further include transmitting data about the BRIR and room characterization. The method may further include performing binaural rendering and individualization at the receiving side based on the determination that the content is transmitted in a listener-independent manner (e.g., does not include HRTFs).The method may further include providing user position and / or orientation data P(t0) at time t1. The uncompressed pre-rendered content may be binauralized uncompressed pre-rendered content. [Brief description of the drawings]

[0037] Embodiments of the present disclosure are described below with reference to the accompanying drawings. [Figure 1] 2 shows a first example of a receiver. [Diagram 2] 2 shows a second example of a receiver. [Diagram 3] 1 shows a first example of a receiver and server system. [Figure 4] 2 shows a second example of a transmitter and receiver system. [Diagram 5] 2 shows a third example of a transmitter and receiver system. [Figure 6] 4 shows a fourth example of a transmitter and receiver system. [Figure 7] The first example shows how media content is processed. [Figure 8] A second example of how media content is processed is shown. [Figure 9] A third example of how media content can be processed is shown below. [Figure 10] A fourth example of how media content is processed is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] As those skilled in the art will appreciate, full immersion in a virtual world "tricks" the brain into believing what is being sensed. When vision is limited by the field of view, sound adds dimension about what is not seen (e.g., a bull charging from behind, a rattlesnake on the right side, or even a whisper moving from the left ear to the back of the head to the right), allowing content creators to use sound to direct the user's gaze and thus effectively tell a story. Currently, movie theaters and home theaters provide immersive audio experiences through object-based or first-order / higher-order Ambisonics (FOA / HOA)-based sound generation, content packaging, and playback. VR sound requires sound precision to fully immerse the user in a virtual world. VR content creators require the ability to generate object-based and / or HOA-based sound in three-dimensional space. Furthermore, such content needs to be encoded, delivered, decoded, and rendered binaurally (through headphones or loudspeakers) with precision and efficiency so that users can enjoy the content.

[0039] The receiver may select a media format representation of content, such as over-the-top (OTT) content delivered via MPEG-DASH or MPEG-MMT formats, based on various parameters, such as bandwidth and media bitrate. The receiver may also receive information regarding media consumption. The selection of the media format representation may be based on such media consumption. For example, pre-rendered binauralized data may be selected based on an indication of headphone or stereo loudspeaker (e.g., with crosstalk cancellation) output.

[0040] The exemplary embodiments described herein describe methods, apparatus, and processes adapted to process media content (e.g., rendering audio content). The exemplary embodiments relate generally to processing media content (e.g., including audio content and / or video content), although, without any intent of limitation, reference is made to audio content in the remainder of this document.

[0041] FIG. 1 illustrates an example of a receiver / client system 100 for binaural rendering. The system 100 can receive an audio input 101. The audio input 101 may include a complete scene contained in an encoded bitstream from a transmitter. The receiver system 100 can receive or detect sensor data (pose information) 110 related to a user's movement and / or a user's head orientation. The sensor data 110 may include information regarding orientation and position, such as, for example, yaw, pitch, roll, and / or (x,y,z) coordinates. The receiver system 100 can further include a decoder 102 that can decode the audio input 101 into uncompressed audio and / or metadata 120. The receiver system 100 can further include a renderer 103 that can render the uncompressed audio and / or metadata 120 into a binaural output 150. The receiver system 100 can output the binaural output 150, for example, to a headphone output.

[0042] The receiver / client system 100 shown in FIG. 1 may suffer from problems related to latency and / or computational complexity as discussed at the beginning of this paper.

[0043] To address these issues, the present disclosure proposes, in a system for processing media content (e.g., audio and / or video content), to obtain posture information about a user at a receiver, transmit the posture information to a transmitter, render the media content based on the posture information, and transmit the rendered media content to the receiver, thereby significantly reducing the computational complexity of the operations performed at the receiver. Furthermore, the rendered media content may be transmitted in uncompressed form, which can reduce the delay between a change in posture (e.g., head movement) and the perception of the reproduced media content adapted to this change in posture (e.g., sound perception), FIG. 7 is a flow chart that generally illustrates an example of a method 700 for processing media content according to the above discussion. The media content may include audio content and / or video content. The audio content may be, for example, FOA-based audio content, HOA-based audio content, object-based audio content, channel-based audio content, or a combination thereof. The processing of the media content may involve rendering the media content. The method may be performed in a system having a first device for reproducing the media content and a second device for providing the media content. Reproducing the media content may involve playing the media content. The first device may be referred to as, for example, a receiver, a receiver device, a client, a client device, or a playback device. The first device may have, correspond to, or operate in association with a computer-mediated reality (e.g., VR, AR, MR) facility, such as, for example, a VR / AR / MR headset (e.g., goggles), and may be associated with a user. The user may wear such a computer-mediated reality facility. The first device may have or be (communicatively) coupled to a sensor (e.g., an attitude sensor) for detecting a posture (e.g., position and / or orientation) of a user or a part of a user (e.g., a user's head). The sensor may further detect a rate of change of posture (first derivative, e.g., velocity, angular velocity(s), yaw / roll / pitch rate(s)). The sensor may further detect a rate of change of the rate of change (second derivative, e.g., acceleration, angular acceleration). The sensor data output by the sensor may be referred to as posture information. In general, it is understood that posture information is indicative of a position and / or orientation (posture) of a user or a part of a user (e.g., a user's head). Additionally, the posture information may be indicative of one or more rates of change of posture (first derivative).Furthermore, the pose information may indicate one or more rates of change (second derivatives), e.g., one or more rates of change of pose. The sensor may be located in alignment with the user or a relevant part of the user (e.g., head), e.g., as part of a computer-mediated reality facility (e.g., VR / AR / MR headset / goggles) or as part of a mobile (computing) device carried by the user (e.g., smartphone, game controller). In this case, the sensor may be referred to as an embedded sensor. Alternatively, the sensor may be provided in or embodied by a location server (e.g., an OptiTrack system or an OptiTrack-type system) that tracks the pose of the user (or part of the user). In general, the sensor may be part of or embodied by a tracking system that tracks the pose of the user (or part of the user). Such a location server may track the poses of multiple users. The second device may be referred to as, for example, a transmitter, a transmitter device, a server, a server device, or a content delivery device. Each of the first and second devices may have a processor (or a set of processors) coupled to a respective memory and adapted (configured) to perform respective operations as described below. For example, the processor (or set of processors) may be adapted (configured) to perform respective steps of method 700 as described below. Alternatively or additionally, the processor (or set of processors) may be adapted (configured) to perform respective steps of any one of methods 800, 900, and 1000 as described below.

[0044] Step S710 In, pose information indicative of a position and / or orientation of a user (or a part of the user, e.g., the user's head) is obtained (e.g., determined). This operation may be performed, for example, by a sensor (e.g., a pose sensor). Step S720 In the latter case, the attitude information is transmitted to a second device. Step S730In the pose information, the media content is rendered based on the pose information to obtain rendered media content. That is, the media content is rendered based on the position and / or orientation of the user or a part of the user. The rendered media content may also be referred to as pre-rendered media content (e.g., pre-rendered audio content and / or pre-rendered video content). If the media content includes audio content, the audio content may be rendered, for example, into binaural audio content, B-format audio content, HOA audio content, channel-based audio content, or a combination thereof. In general, the audio content may be rendered into two or more channels and / or components. If the media content includes video content, the video content may be tiled and a region of interest of an overall video scene may be output, for example, as rendered video content. Step S740 In step S710, the rendered media content is transmitted to the first device for reproduction. Steps S710 and S720 may be performed in / by the first device, e.g., by a sensor (e.g., an orientation sensor) and a (first) transmitting unit, respectively. Steps S730 and S740 may be performed in / by the second device, e.g., by a renderer and a (second) transmitting unit.

[0045] For audio content, method 700 may further include generating an audible (e.g., acoustic) representation of the rendered audio content, e.g., via two or more loudspeakers that are part of or coupled to the first device. The two or more loudspeakers may, for example, be part of a computer-mediated reality facility. For video content, method 700 may further include generating a visual representation of the rendered video content, e.g., via a display device that is part of or coupled to the first device. The display device may, for example, be part of a computer-mediated reality facility. In general, generation of such a representation may be performed in / by the first device.

[0046] An example of a receiver / client system 200 for binaural rendering according to the above-mentioned method is shown diagrammatically in FIG. 2. The system may embody the first device of the method 700. The system 200 may receive an audio input 201 as an example of rendered media content (rendered audio content). The audio input 201 may be, for example, in the form of binauralized uncompressed audio. The receiver system 200 may output sensor data (as an example of pose information) relating to the movement of a user and / or the orientation of the user's head. The head pose / sensor data 220 may include, for example, information regarding yaw, pitch, roll and / or (x,y,z) coordinates. The receiver system 200 may output the sensor data to a sender / server. The sender / server may embody the second device of the method 700. The receiver system 200 may further generate an audible representation of the audio input 201. For example, the receiver system may output the uncompressed audio input 201 to a headphone output.

[0047] As will be described in more detail below, any of the systems shown in Figures 3, 4, 5 and 6 may implement the method 700.

[0048] To further reduce delays between a change in posture and a corresponding adaptation of a representation of the media content presented to the user, the second device may predict posture information to anticipate possible delays resulting from transmission to the first device and / or encoding / decoding (described below). For example, rendering the media content in step S730 of method 700 may involve obtaining (e.g., determining, calculating) predicted posture information and rendering the media content based on the predicted posture information (rather than based on posture information received from the first device).

[0049] 8 is a flow chart that generally illustrates an example of a method 800 for processing media content applying pose information prediction. Unless otherwise stated, statements made in relation to method 700 above apply here too.

[0050] Step S810 and Step S820 corresponds to steps S710 and S720, respectively, of method 700. Step S830aIn step S820, predicted pose information is obtained (e.g., determined, calculated) based on the pose information received in step S820 and the previous pose information. If the pose information includes first and / or second derivatives of the pose, the prediction can be based on said first and / or second derivatives in addition to or instead of the previous pose information. The predicted pose information may be pose information for a future time point, e.g., indicative of a position of the user and / or a part of the user (e.g., head) at a future time point. In certain implementations, the predicted pose information may be predicted for an estimate of a time point at which the rendered media content is expected to be processed by the first device for reproduction. The estimate of a time point at which the first device is expected to process the rendered media content for reproduction may include an estimate of a time (duration) required to transmit the rendered media content to the first device. Alternatively or additionally, if encoding / decoding (e.g., compression / decompression) is applied (described below), the estimate of the time point may include an estimate of a time (duration) required to encode / decode the rendered media content. That is, the predicted attitude information may be obtained further based on an estimate of the time required to transmit the rendered media content and / or the time required to encode / decode the rendered media content. The previous attitude information may be attitude information received from the first device at a previous time. One or more items of the previous attitude information may be used to obtain the predicted attitude information, for example via extrapolation or model-based prediction techniques. For this purpose, an item (e.g., a predetermined number of items) of the previous attitude information may be stored. Step S830b In step S720, the media content is rendered based on the predicted pose information to obtain rendered media content. This operation may differ from step S730 of method 700 in that the predicted pose information is used instead of the pose information (received in step S720 or step S820), but may otherwise be performed in the same manner as step S730. Step S840In the method, the rendered media content is transmitted to a first device for reproduction. Steps S810 and S820 may be performed in / by the first device. Steps S830a, S830b, and S840 may be performed in / by a second device. Step S830a may be performed, for example, by a pose predictor.

[0051] For audio content, the method 800 may further include generating an audible (e.g., acoustic) representation of the rendered audio content, e.g., via two or more loudspeakers that are part of or coupled to the first device. The two or more loudspeakers may, for example, be part of a computer-mediated reality facility. For video content, the method 800 may further include generating a visual representation of the rendered video content, e.g., via a display device that is part of or coupled to the first device. The display device may, for example, be part of a computer-mediated reality facility. In general, the generation of such a representation may be performed in / by the first device.

[0052] In one modification of method 800, the predicted attitude information may be predicted in the first device. That is, the first device may perform processing as described above with reference to step S830a and then transmit the predicted attitude information to the second device. In this case, it is understood that step S820 may be omitted. After receiving the predicted attitude information from the first device, the second device may continue its processing in step S830b and subsequent steps in the manner described above.

[0053] As will be described in more detail below, any of the systems shown in Figures 3, 4, 5 and 6 may implement the method 800 or the above modifications of method 800.

[0054] The prediction of pose information for rendering the media content as described above allows to "hide" the delays caused by the transmission and / or encoding / decoding such that a better alignment between the user's movements and the presentation of the rendered media content can be achieved. Thus, the risk of the user being affected by motion sickness can be reduced or avoided altogether and the user's immersive computer-mediated reality experience can be improved. In the case of the method 800, the improved alignment between the movements and the presentation of the rendered media content is achieved by a process performed at the server / sender, i.e., by predicting the pose information and using the predicted pose information instead of the pose information received from the receiver / playback side for rendering the media content. However, under certain conditions, it may be desirable to achieve such an improved alignment between the movements and the presentation of the rendered media content by measures performed at the receiver or playback side.

[0055] FIG. 9 is a flow chart that illustrates generally an example of a method 900 for processing media content in accordance with the above discussion, i.e., improving alignment between motion and the presentation of the rendered media content by measures performed at the receiver / playback side.

[0056] Step S910 , Step S920 , Step S930 , and Step S940 correspond to steps S710 to S740 of method 700, respectively. Step S950 In the method, pose information used to render the media content (e.g., pose information received from the first device) is transmitted to the first device, and the pose information may be transmitted together with, e.g., in association with, the rendered media content. Step S960In step S710, the orientation information used to render the media content is compared to current orientation information, which may be orientation information obtained when reproducing (e.g., playing) the rendered media content. The current orientation information may be obtained in the manner described above with reference to step S710, but at a different (later) time. Step S970 In the method, the rendered media content is updated based on the results of the comparison. For example, the rendered media content may be updated based on a difference between the pose information used to render the media content and the current pose information. The updating may involve extrapolation of the rendered media content. Non-limiting examples of such updating are described below with reference to FIG. 3. Steps S910, S920, S960 and S970 may be performed at / by the first device. Steps S930, S940 and S950 may be performed at / by the second device.

[0057] In some implementations, step S950 may be omitted, i.e., the pose information used to render the media content may not be sent to the first device, in which case step S960 may refer to the pose information sent to the second device in step S920 as the pose information used to render the media content.

[0058] Additionally, in certain implementations, the method 900 may include determining gradient information of how the rendered media content changes in response to changes in pose information (e.g., in response to changes in the pose of the user or the pose of the user's head). The method 900 may then further include transmitting the gradient information to the first device. For example, the gradient information may be transmitted to the first device together with (e.g., in association with) the rendered media content and optionally with the pose information used to render the media content. These additional steps may be performed in the second device. For audio content, the gradient information may indicate changes in sub-band energy levels (e.g., for each channel or component) in response to translation and / or rotation of the user or a portion of the user. The gradient information may then be used to update / adjust the rendered media content in step S970. For example, the sub-band energy levels of the rendered audio content may be adjusted based on the gradient information and a difference between the pose information used to render the media content and the current pose information. In general terms, the rendered media content can be updated / adjusted based on the pose difference and a gradient that indicates the change in the rendered media content in response to the pose change.

[0059] For audio content, the method 900 may further include generating an audible (e.g., acoustic) representation of the rendered audio content, e.g., via two or more loudspeakers that are part of or coupled to the first device. The two or more loudspeakers may, for example, be part of a computer-mediated reality facility. For video content, the method 900 may further include generating a visual representation of the rendered video content, e.g., via a display device that is part of or coupled to the first device. The display device may, for example, be part of a computer-mediated reality facility. In general, the generation of such a representation may be performed in / by the first device.

[0060] As will be described in more detail below, any of the systems shown in Figures 3, 4, 5 and 6 may implement the method 900.

[0061] To further improve the alignment between the user's movements and the presentation of the rendered media content, prediction of pose information at the server / sender and updating of the rendered media content at the receiver / playback may be combined.

[0062] FIG. 10 is a flow chart that generally illustrates an example of a method 1000 for processing media content in accordance with the above discussion, i.e., improving alignment between motion and presentation of the rendered media content by measures performed at the server / sender side and measures performed at the receiver / playback side.

[0063] Step S1010 , Step S1020 , Step S1040 correspond to steps S710, S720, and S740, respectively, of method 700. Step S1030a and Step S1030b corresponds to steps S830 and S830b, respectively, of method 800. Step S1050In the method, the predicted pose information (i.e., the pose information used to render the media content) is transmitted to the first device. The predicted pose information may be transmitted together with the rendered media content, e.g., in association with the rendered media content. Step S1060 In step S710, the predicted pose information is compared to actual / current pose information, which may be pose information obtained when reproducing (e.g., playing) the rendered media content, or may be obtained in the manner described above with reference to step S710, but at a different (later) time. Step S1070 In the method, the rendered media content is updated based on the results of the comparison. For example, the rendered media content may be updated based on a difference between the predicted pose information and the actual pose information. In general, the updating may be performed in the same manner as step S970 of method 900. Steps S1010, S1020, S1060 and S1070 may be performed in / by the first device. Steps S1030a, S1030b, S1040 and S1050 may be performed in / by the second device.

[0064] In certain implementations, the method 1000 may include determining gradient information of how the rendered media content changes in response to changes in the pose information (e.g., in response to changes in the pose of the user or the pose of the user's head). The method 1000 may then further include transmitting the gradient information to the first device. For example, the gradient information may be transmitted to the first device together with (e.g., in association with) the rendered media content and optionally with the pose information used to render the media content. These additional steps may be performed in the second device. For audio content, the gradient information may indicate changes in sub-band energy levels (e.g., of each channel or component) in response to translation and / or rotation of the user or a part of the user. The gradient information may then be used to update / adjust the rendered media content in step S1070. For example, the sub-band energy levels of the rendered audio content may be adjusted based on the gradient information and a difference between the pose information used to render the media content and the current pose information. In general terms, the rendered media content may be updated / adjusted based on the pose difference and a gradient that indicates the change in the rendered media content in response to the change in pose.

[0065] For audio content, method 1000 may further include generating an audible (e.g., acoustic) representation of the rendered audio content, e.g., via two or more loudspeakers that are part of or coupled to the first device. The two or more loudspeakers may, for example, be part of a computer-mediated reality facility. For video content, method 1000 may further include generating a visual representation of the rendered video content, e.g., via a display device that is part of or coupled to the first device. The display device may, for example, be part of a computer-mediated reality facility. In general, the generation of such a representation may be performed in / by the first device.

[0066] In a modification of the method 1000, the predicted orientation information may be predicted in the first device. That is, the first device may perform the process described above with reference to step S1030a and then transmit the predicted orientation information to the second device. It is understood that in this case, step S1020 may be omitted. After receiving the predicted orientation information from the first device, the second device may use the predicted orientation information to render the media content in the manner described above with reference to step S1030b and transmit the rendered media content to the first device in the manner described above with reference to step S1040. In this case, step S1050 may be omitted. After receiving the rendered media content, the first device may perform steps S1060 and S1070 in the manner described above. Notably, in this case, since the prediction of the orientation information is performed in the first device, the first device does not need to receive the predicted orientation information from the second device.

[0067] As will be described in more detail below, any of the systems shown in Figures 3, 4, 5 and 6 may implement the method 1000 or the above-described modifications of the method 1000.

[0068] In any of the above methods, the rendered media content may be transmitted to the first device in uncompressed form. This is made possible by pre-rendering in the second device and does not require the transmission of the full media content (e.g., a complete representation of an audio / video scene). Transmitting the rendered media content in uncompressed form contributes to reducing the round-trip delay since the time normally spent on compression / decompression is saved. On the other hand, the rendered media content may be encoded (compressed) before transmission to the first device if required by bandwidth limitations. In this case, the time required for encoding / decoding (e.g., compression / decompression) may be taken into account when obtaining the predicted pose information, as described above.

[0069] Further, for audio content, any of the above methods may further include transmitting environmental information to the second device indicative of acoustic characteristics of an environment in which the first device is located. The environmental information may include room characteristics and / or a Binaural Room Impulse Response (BRIR) function. This step may be performed in / by the first device, e.g., at set-up. The audio content may then be rendered further based on the environmental information. Alternatively or additionally, any of the above methods may further include transmitting morphological information indicative of a morphology of the user or a part of the user to the second device. The morphology may include or correspond to a shape or size, e.g., a shape or size of the user's head. The morphological information may include a Head Related Transfer Function (HRTF). The rendering may be a binaural rendering. This step may be performed in / by the first device, e.g., at set-up. The audio content may then be rendered further based on the morphological information.

[0070] FIG. 3 shows further details of an exemplary system including a server / sender 300 and a client / receiver 350. As described above, the system may implement any of the methods 700, 800, 900, and 1000. The server / sender 300 (e.g., implementing the second device) may include a renderer 320 (e.g., an audio renderer) and an encoder 330. The client / receiver 350 (e.g., implementing the first device) may send a current pose (e.g., head pose) P(t0) at time t0 to the server / sender 300. The current pose P(t0) may also include a timestamp t0 itself that specifies the time when the current pose P(t0) was generated. The pose P(t0) may be determined and sent by the pose 350 block.

[0071] The server / transmitter 300 (e.g., implementing a second device) may further include a position predictor 310. The server / transmitter 300 may receive a user position and a current pose P(t0) (corresponding to the head orientation) at time t1 (where t1>t0). The received current pose P(t0) and t0 itself may be used by the position predictor 310 to predict the position P(t1). The position predictor 310 may take into account previously received poses P(tn) and tn, where n may be 0 to -infinity (pose and timestamp values ​​from earlier time points) to predict the position P(t1). The position P(t1) may be similar to the pose P(t0). The position P(t1) may be used by the audio renderer 320 to render the audio scene at time t1, thereby determining the rendered audio data R(t1) 340. The rendered audio data R(t1) 340 may be encoded using an audio encoder 330 to determine audio data A(t1). The server / transmitter 300 may transmit the audio data A(t1) and the position P(t1) to the client / receiver 350. The position P(t1) may be encoded as part of the audio bitstream. The client / receiver 350 may receive the audio data A(t1) and the position P(t1) (e.g., in the form of metadata) from the server / transmitter 300 at time t2, where t2>t1. The client / receiver 350 may receive the audio data A(t1) and the position P(t1) at an audio decoder 351, which may determine the uncompressed audio U(t1). The head pose / sensor data 352 block may determine the pose P(t2) at time t2. The audio extrapolator 353 can use the received P(t1) to calculate a posture difference DeltaP by subtracting posture P(t1) from posture P(t2) at time instant t2. DeltaP may be used by the audio extrapolator 353 to adapt / extrapolate the uncompressed audio U(t1) before outputting 390.The client / receiver 350 may apply local rotation as part of the extrapolation if the audio content is FOA and its motion is constrained to yaw, pitch and / or roll motions. The client / receiver 350 may further apply a blind upmix as part of the extrapolation if the audio content is pre-rendered binaural content or pre-rendered channel-based content.

[0072] Instead of predicting the position P(t1), the position P(t2') may be predicted for a time t2' when the client / receiver 350 is expected to receive or process the audio data. The time t2' may be estimated taking into account the time (duration) required for transmitting and / or encoding / decoding the audio data, starting from the time t1. P(t1), R(t1), A(t1) and U(t1) above must then be replaced by P(t2'), R(t2'), A(t2') and U(t2'), respectively. Any of the above elements may be implemented by a processor (or a set of processors) of the respective device.

[0073] The following syntax from MPEG-H 3D Audio (ISO / IEC23008-3) and / or future versions of the MPEG standard may be used to transmit the 3 Degrees of Freedom (3DoF) data of P(t). [Table 1]

[0074] The semantics may be defined according to MPEG-H 3D Audio (ISO / IEC23008-3) and / or future versions of the MPEG standard.

[0075] The complete syntax for sending 6DoF data and timestamps may look like this: [Table 2]

[0076] The semantics may be defined according to MPEG-H 3D Audio (ISO / IEC23008-3) and / or future versions of the MPEG standard.

[0077] FIG. 4 illustrates an exemplary system including a sender 400 and a receiver 450. The system described in FIG. 4 may include, in whole or in part, aspects of the systems described in FIG. 1-3. In particular, the system may implement any or all of the methods 700, 800, 900, and 1000 described above. A sender / server 400 (e.g., implementing a second device) may receive a complete audio scene (e.g., FOA / HOA or object-based scene) 401 and a complete video scene (e.g., 360° video) 402 (as examples of media content). The audio scene 401 may be processed by an audio encoder 410 to determine a complete audio scene delivery 441. The complete audio scene 441 may consist of the entire audio scene and / or associated metadata (audio object position, orientation, etc.). The complete video 402 may be processed by a content delivery selection 420. The complete video 402 may be divided into various parts, such as regions of interest, and "tiled" accordingly by the content selection 420 (a 360° video may be divided into tiles) to determine the tiles 402a. The content delivery and selection 420 may use the predicted position P(t1) (or the predicted position P(t2')), illustrated as an output from the position predictor 310 in FIG. 3, or may use the unmodified head pose / sensor data 454. For example, the tile 402a of the complete 360° video 402 may be selected in the content delivery selection 420 based on the sensor data 454 received from the receiver 450. This selection may be referred to as rendering the video content. The video encoder 430 encodes the tiles 402a to output a region of interest video 442 that may be transmitted to a client / receiver 450 (e.g., implementing a first device). The receiver 450 may include a video decoder 452 capable of receiving the region of interest video 442. The video decoder 452 can decode the video using the region of interest 442 and output it to a video output 492 .The complete audio scene 441 may be received by an audio decoder 451, which may decode the content and provide the decoded audio scene to an audio renderer 453. The audio decoder 451 may provide uncompressed audio and metadata 455, which may correspond to the decoded audio scene, to the audio renderer 453. The audio renderer 453 may render the decoded audio based on sensor data 454 and may output an audio output 491. The sensor data 454 may be received from a sensor (e.g., a gyroscope-based sensor) that may detect the user's movement and / or the user's head orientation. It may then be provided to the audio renderer 453 to adapt the complete audio scene 441 to the user's current head orientation and / or position, and to the content delivery selection 420 to adapt the complete video scene 402 to the user's current head orientation and / or position. In particular, in the exemplary system of Figure 4, video content is rendered at the server / sender (i.e., video content is generated at the server / sender ready for playback at the receiver / playback), and audio content is rendered at the receiver / playback. Any of the above elements may be implemented by a processor (or a set of processors) of the respective device.

[0078] FIG. 5 illustrates an exemplary system including a transmitter 500 and a receiver 550. The system may implement any or all of the methods 700, 800, 900, and 1000 described above. A transmitter / server 500 (e.g., implementing a second device) may receive a complete audio scene (e.g., HOA or object-based scene) 501 and a complete video scene (e.g., 360° video) 502 (as examples of media content). The audio scene 501 may be processed by an audio renderer 510 to determine binaural audio data 541. The audio renderer 510 may determine the binaural audio data 541 taking into account sensor data 545. The sensor data 545 may include yaw, pitch, roll, x, y, z information. The binaural audio data 541 may be uncompressed, lossless compressed, or lossless low-latency compressed. For example, the binaural audio data 551 may be uncompressed audio 580 that may be received by a receiver 550 (e.g., implementing a first device) and provided to a binaural audio output 591. The complete video 502 may be processed by a content delivery selection 520. The complete video 502 may be divided into various parts, such as regions of interest, and "tiled" accordingly (the 360° video is divided into tiles) to determine tiles 502a in the content delivery selection 520. The tiles 502a of the complete 360° video 502 may be selected in the content delivery selection 520 based on sensor data 545 received from the receiver 550. This selection may be referred to as rendering the video content. The video encoder 530 encodes the tiles 502a to output a video of the region of interest 542 that may be transmitted to the client / receiver 550. The receiver 550 may include a video decoder 552 that may receive the video of the region of interest 542. The video decoder 552 can use the region of interest 542 to decode the video and output it to a video output 592 .The sensor data 545 may be received from a sensor (e.g., a gyroscope-based sensor) capable of detecting the user's movements and / or the user's head orientation. It may then be further provided to the content delivery selection 520 to adapt the complete video scene 502 to the user's current head orientation and / or position. It may then be provided to the content audio renderer 510 to adapt the complete audio scene 501 to the user's current head orientation and / or position. Any of the above elements may be implemented by a processor (or a set of processors) of the respective device.

[0079] FIG. 6 illustrates an exemplary system including a transmitter 600 and a receiver 650. The system may implement any or all of the methods 700, 800, 900, and 1000 described above. The transmitter / server 600 (e.g., implementing the second device) may receive a complete audio scene (e.g., HOA or object-based scene) 601 and a complete video scene (e.g., 360° video) 602 (as examples of media content). The audio scene 601 may be processed by an audio renderer 610, the output of which may then be processed by a low-delay audio encoder 660. The audio renderer 610 may take into account sensor data 645. The low-delay audio encoder 660 may output binaural audio data 641, which may then be sent to the receiver 650 (e.g., implementing the first device). The binaural audio data 641 is received at the receiver 650 by a low-delay audio decoder 670, which converts the binaural audio data 641 to uncompressed audio 680. The uncompressed audio 680 may then be provided to a binaural audio output 691. The complete video 602 may be processed by a content delivery selection 620. The complete video 602 may be divided into various parts such as regions of interest in the content delivery selection 620 and "tiled" accordingly (a 360° video may be divided into tiles) and may determine the tiles that may be selected in the content delivery selection 620 based on the sensor data 645 received from the receiver 650. This selection may be referred to as rendering the video content. The video encoder 630 encodes the tiles and / or video to output a region of interest video 642 that may be transmitted to the client / receiver 650. The receiver 650 may include a video decoder 652 that can receive the region of interest video 642. The video decoder 652 may decode and output the video using the region of interest 642 to a video output 692.The sensor data 645 may be received from a sensor (e.g., a gyroscope-based sensor) capable of detecting the user's movements and / or the user's head orientation. It may then be further provided to the Content Delivery Selection 620 to adapt the complete video scene 602 to the user's current head orientation and / or position. It may then be further provided to the Content Audio Renderer 610 to adapt the complete audio scene 601 to the user's current head orientation and / or position. Any of the above elements may be implemented by a processor (or a set of processors) of the respective device.

[0080] Typically, audio (as a non-limiting example of media content) transmitted from a sender (S) to a receiver (R) is rendered at the receiver as shown in Figures 1 and 4. To maximize flexibility at the receiver side, it is possible to send complex representations of the audio scene, such as objects or HOAs, which can be adaptively rendered at the receiver side, for example to fit the local listener's viewpoint / pose. However, the large latency that encoding such representations may require prevents these approaches from being used for communication or interactive applications.

[0081] The present disclosure provides methods, systems, and devices for reducing the above-mentioned latency and / or reducing the computational complexity at the receiver. The user position and orientation transmitted from the receiver to the sender allows the server / sender to compute a more compact, pre-rendered version of the content that closely matches the receiver's current pose / viewpoint. However, the transmission latency from the sender to the receiver may introduce a mismatch between the local receiving pose and the pose at which the rendering was computed on the server. The present disclosure proposes that the sender signal the position at which the rendering was performed so that the receiver can extrapolate the rendered signal to the current local pose. In addition, the sender can transmit a pre-rendered, uncompressed or lossless compressed representation of the audio scene to eliminate encoding and decoding latencies in the system. The sender executes a rendering algorithm, e.g. binaural stereo, FOA or HOA. The rendering algorithm can render audio data such as audio objects (e.g. pre-rendered binauralized content) into two channels to an output channel. Those channels may then be encoded, especially if compression is required (e.g., depending on the system bandwidth), to output an encoded audio data bitstream. The signal can be sent to a client or receiver and output via headphones or a stereo loudspeaker system.

[0082] When the binauralized reproduction needs to be adapted to the physical characteristics of the user's head, the receiver can transmit a Head-Related Transfer Function (HRTF) corresponding to the characteristics of the user's head. The receiver can further transmit a Bilateral Room Impulse Response (BRIR) function corresponding to the room that is intended to be reproduced. This information may be transmitted during the transmission setup.

[0083] Embodiments of the present disclosure may provide at least the following advantages: If only binauralized (stereo) data is transmitted, the transmission bitrate for lossless audio data transmission may be similar or equal to the bitrate for the complete compressed audio scene. Transmitting uncompressed or lossless audio streams eliminates or reduces the latency due to encoding and decoding. For example, the latency resulting from encoding / decoding may be reduced to zero, resulting in an overall reduction in motion-to-ear latency. ● In the absence of compression of the audio data, the receiver simply outputs the audio without decoding or rendering. This results in reduced computational complexity at the receiver since there is no need to perform decoding and / or rendering has already been completed at the sender. Various trade-offs may be adopted between minimal latency and minimal receiver computational complexity at higher bit rates and minimal bit rates at higher latency and higher receiver computational complexity. For example: ○ Transmission of uncompressed data for minimal latency and computational complexity, but sufficient bandwidth to transmit uncompressed data Transmission of losslessly compressed data for minimal latency and slightly higher computational complexity when bandwidth is not sufficient to transmit uncompressed data Transmission of lossy but low-latency compressed data for low latency and higher computational complexity when bandwidth is limited.

[0084] The above also applies to bidirectional communication between R and S when R and S are simultaneously receivers and transmitters.

[0085] Table 3 shows an example system comparison that illustrates such trade-offs. [Table 3]

[0086] In certain contexts, embodiments of the present disclosure may involve hiding transmission latency based on content extrapolation. If the overall latency (e.g., transmission latency) is too high (typically greater than 20 msec), it is desirable for the encoding format and / or playback system to provide a means to extrapolate content to match the local attitude (position and orientation) of the receiver while waiting for delivery of the next updated content frame. The overall latency may be determined based on the sum of all latencies in the round trip of audio data. For example, the overall latency may be based on the round trip latency, the encoding latency, the decoding latency, and the rendering latency.

[0087] Hiding this latency can be achieved by sending a local pose from the receiver to the sender / server for rendering (e.g., as described above with reference to steps S920 and S1020) and having the sender / server send back which pose was used for each rendered frame of the content (e.g., as described above with reference to steps S950 and S1050). The sender / sender can predict the user's movements, including taking into account previously received positions, to compensate for the additional latency introduced between when the content is rendered by the sender and when it is received at the receiver.

[0088] The receiver can then extrapolate the pre-rendered audio received from the server given the delta between the posture used to render the content at the sender (e.g., as described above with reference to steps S970 and S1070) and the local posture of the receiver R (e.g., current or actual posture).

[0089] This extrapolation can be implemented in several ways based on the flexibility of the rendered content. In one example, when the content is pre-rendered Ambisonics B format and the motion is 3-DOF motion, the extrapolation may be based on a client-side local rotation of the FOA or B format content before playback. In another example, for pre-rendered binaural content, the extrapolation can be achieved by a blind upmix (see Annex A) or by adding metadata to the binaural stream (see Annex B). In another example, for pre-rendered channel-based content, a low-latency blind upmixer can be applied at the receiving end.

[0090] If rendering and encoding are tightly integrated at the sender, it is possible to increase the flexibility of the pre-rendered content by adding metadata encoding, e.g., the direction and / or energy gradients ∇E(P) of different sub-bands based on the current rendering position P.

[0091] If the original content to be rendered is object-based, it is possible to compute multiple renderings around the desired position and encode a level gradient. This level gradient G typically consists of a 3D vector (one value for each of the three axes x, y, z). The receiver can then simply adjust the subband energy E(P) in the received signal based on the difference between the pre-rendered position P and the current receiver position P', as E(P') = E(P) · (P'-P) · ∇E(P).

[0092] This extra information may be used by the receiver to further extrapolate the pre-rendered stream (i.e., pre-rendered media content), for example by taking into account parallax effects (using distance information) or adjusting the level of rendering (using level gradient information).

[0093] In one example, if the receiver is constrained in terms of computational power, the upmix may be performed during encoding at the transmit side, for example B-format or channels may be converted to objects. This may increase the encoding path latency, but the resulting content may be more flexible and may be extrapolated at the receive end.

[0094] For gaming applications, where user actions (e.g. button triggers) can affect game play, the overall system latency still needs to be less than 20msec, which may prevent complex upmix operations from being performed. As a result, flexible formats like B-format may be the best candidates for rendering / transmission using low-latency lossless or lossy codecs, since they can be rendered and rotated with low latency at the receiving end as well.

[0095] Various audio codecs can incorporate the above mentioned data transmission modes. The codec may be adapted for: (i) the possibility to transmit losslessly coded (zero latency coded) stereo audio data or low latency lossy data; (ii) a means to signal that the content is already pre-rendered (e.g. a bitstream syntax field; Dolby AC-4 and MPEG-H Part 3, 3D audio already contain such a bitfield; e.g. b_pre_virtualized in Dolby AC-4) if "normal" rendering (e.g. binauralization in the device) needs to be turned off; (iii) a means to transmit HRTF and BRIR if necessary.

[0096] Thus, in the context of the present disclosure, the sender may also provide an indication (e.g., a flag, a bit field, a syntax field / element, a parameter) to the receiver to provide pre-rendered audio content. If such an indication is received by the receiver, the receiver may refrain from any (receiver-side) rendering of the audio content. For example, for binaural pre-rendered audio content, the receiver may route the pre-rendered audio content received from the sender directly to the headphones (speakers) for reproduction without further rendering. Such an indication may be in the form of a parameter directHeadphone signaled in the bitstream to the receiver. The directHeadphone parameter may define that the corresponding signal group of the (type) channel goes directly to the headphone output if a binaural output is rendered. The signals may be routed to the left and right headphone channels.

[0097] A possible example of the syntax for this parameter is reproduced in Table 4. [Table 4]

[0098] The semantics may be defined according to MPEG-H 3D Audio (ISO / IEC 23008-3) and / or future versions of the MPEG standard.

[0099] Further exemplary embodiments of the present disclosure are summarized in the enumerated example embodiments (EEE) listed below.

[0100] EEE1 is 1. A method of processing media content for reproduction by a first device, comprising: obtaining attitude information indicative of a position and / or orientation of a user; transmitting the orientation information to a second device providing the media content; rendering the media content based on the pose information to obtain rendered media content; transmitting the rendered media content to the first device for reproduction. It concerns the method. EEE2 is the media content comprises audio content and the rendered media content comprises rendered audio content; and / or the media content comprises video content, and the rendered media content comprises rendered video content; This relates to the method described in EEE1. EEE3 is the media content includes audio content, and the rendered media content includes rendered audio content, the method further including generating an audible representation of the rendered audio content. This relates to the method described in EEE1. EEE4 is The method of claim 8, wherein the audio content is one of First Order Ambisonics (FOA) based, Higher Order Ambisonics (HOA) based, object based or channel based audio content or a combination of two or more of FOA based, HOA based, object based or channel based audio content. EEE5 is The method of any one of claims 2 to 3, wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content or channel-based audio content or a combination of two or more of binaural audio content, FOA audio content, HOA audio content or channel-based audio content. EEE6 is The rendering is: obtaining predicted attitude information based on the attitude information and previous attitude information; and rendering the media content based on the predicted pose information to obtain the rendered media content. The present invention relates to a method according to any one of claims EEE1 to EEE5. EEE7 is transmitting the predicted pose information to the first device along with the rendered media content. This relates to the method described in EEE6. EEE8 is comparing the predicted attitude information with actual attitude information; and updating the rendered media content based on the results of the comparison. This relates to the method described in EEE7. EEE9 is the predicted pose information is predicted for an estimate of a time at which the rendered media content is expected to be processed by the first device for reproduction; the actual pose information being pose information obtained at the time the rendered media content is actually processed for reproduction by the first device; This relates to the method described in EEE8. EEE10 is The method of any one of claims 8 to 10, wherein the rendered media content is transmitted to the first device in an uncompressed form. EEE11 is encoding the rendered media content prior to transmission to the first device, and decoding the encoded rendered media content after receipt at the first device. The present invention relates to a method according to any one of claims EEE1 to EEE10. EEE12 is The method of any EEE method including the subject matter of EEE9 or EEE9, wherein the estimate of a time at which the rendered audio content is expected to be processed by the first device for reproduction comprises an estimate of a time required to encode and decode the rendered audio content and / or an estimate of a time required to transmit the rendered media content to the first device. EEE13 is The method of any EEE including the subject matter of EEE6 or EEE7, wherein the predicted attitude information is obtained further based on an estimate of a time required to encode and decode the rendered media content and / or an estimate of a time required to transmit the rendered media content to the first device. EEE14 is comparing pose information used to render the media content with current pose information; and updating the rendered media content based on the results of the comparison. The present invention relates to a method according to any one of claims EEE1 to EEE13. The EEE15 is determining, at the second device, gradient information indicative of how the rendered media content changes in response to changes in the pose information; transmitting the gradient information along with the rendered media content to the first device; and comparing, at the first device, pose information used to render the media content with current pose information. and updating the rendered media content based on the gradient information and the comparison. The present invention relates to a method according to any one of claims EEE1 to EEE14. EEE16 is the media content comprises audio content, and the rendered media content comprises rendered audio content; The method further includes transmitting environmental information to the second device, the environmental information being indicative of an acoustic characteristic of an environment in which the first device is located; the rendering of the media content is further based on the environmental information. The present invention relates to a method according to any one of claims EEE1 to EEE15. EEE17 is the media content comprises audio content, and the rendered media content comprises rendered audio content; The method further includes transmitting morphological information indicative of a morphology of a user or a portion of a user to the second device; the rendering of the media content is further based on the morphological information; The present invention relates to a method according to any one of claims EEE1 to EEE16. EEE18 is 1. A system having a first device for reproducing media content and a second device for storing said media content, comprising: The first device comprises: Obtaining attitude information indicating the user's position and / or orientation; · adapted to transmit the attitude information to the second device; The second device comprises: rendering the media content based on the pose information to obtain rendered media content; adapted to transmit the rendered media content to the first device for reproduction; Regarding the system. EEE19 is the media content comprises audio content and the rendered media content comprises rendered audio content; and / or the media content comprises video content, and the rendered media content comprises rendered video content; This relates to the system described in EEE18. The EEE20 is the media content comprises audio content, and the rendered media content comprises rendered audio content; the first device is further adapted to generate an audible representation of the rendered audio content. This relates to the system described in EEE18. EEE21 is The system described in EEE19 relates to a system in which the audio content is one of First Order Ambisonics (FOA) based, Higher Order Ambisonics (HOA) based, object based or channel based audio content or a combination of two or more of FOA based, HOA based, object based or channel based audio content. EEE22 is The system of any one of claims EEE19 to 21, wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content or channel-based audio content or a combination of two or more of binaural audio content, FOA audio content, HOA audio content or channel-based audio content. EEE23 is, The second device further comprises: obtaining predicted attitude information based on the attitude information and previous attitude information; and adapted to render the media content based on the predicted pose information to obtain the rendered media content. The present invention relates to a system according to any one of claims EEE18 to 22. EEE24 is the second device further comprising: and adapted to transmit the predicted pose information to the first device together with the rendered media content. This relates to the system described in EEE23. EEE25 is the first device further comprising: Comparing the predicted attitude information with the actual attitude information; and further adapted to update the rendered media content based on a result of the comparison. This relates to the system described in EEE24. EEE26 is the predicted pose information is predicted for an estimate of a time at which the rendered media content is expected to be processed by the first device for reproduction; the actual pose information being pose information obtained at the time the rendered media content is actually processed for reproduction by the first device; This relates to the system described in EEE25. EEE27 is The system of any one of EEE18 to 26, wherein the rendered media content is transmitted to the first device in uncompressed form. EEE28 is the second device is further adapted to encode the rendered media content prior to transmission to the first device; the first device is further adapted to decode the encoded rendered media content after receipt at the first device. The system according to any one of claims EEE18 to 27. EEE29 is The system described in any of EEE including subject matter EEE26 or EEE26, wherein the estimate of the time at which the rendered audio content is expected to be processed by the first device for reproduction includes an estimate of the time required to encode and decode the rendered audio content and / or an estimate of the time required to transmit the rendered media content to the first device. The EEE30 is The present invention relates to a system as described in any of EEE including the subject matter of EEE23 or EEE23, wherein the predicted pose information is obtained further based on an estimate of a time required to encode and decode the rendered media content and / or an estimate of a time required to transmit the rendered media content to the first device. EEE31 is the first device further comprising: comparing pose information used to render the media content with current pose information; and adapted to update the rendered media content based on a result of the comparison. This relates to a system as described in any one of claims EEE18 to 30. EEE32 is the second device further comprising: determining gradient information indicative of how the rendered media content changes in response to changes in the pose information; adapted to transmit the gradient information to the first device together with the rendered media content; the first device further comprising: comparing pose information used to render the media content with current pose information; and updating the rendered media content based on the gradient information and a result of the comparison. The present invention relates to a system according to any one of claims EEE18 to 31. EEE33 is the media content comprises audio content, and the rendered media content comprises rendered audio content; The first device is further adapted to transmit environmental information indicative of an acoustic characteristic of an environment in which the first device is located to the second device; the rendering of the media content is further based on the environmental information. The present invention relates to a system as described in any one of claims 8 to 32. EEE34 is the media content comprises audio content, and the rendered media content comprises rendered audio content; The first device is further adapted to transmit morphological information indicative of a morphology of a user or a part of a user to the second device; the rendering of the media content is further based on the morphological information; The present invention relates to a system according to any one of claims EEE18 to EEE33. EEE35 is A second device providing media content for reproduction by the first device, the second device comprising: receiving attitude information indicative of a position and / or orientation of a user of the first device; rendering the media content based on the pose information to obtain rendered media content; adapted to transmit the rendered media content to the first device for reproduction. Regarding the second device. EEE36 is the media content comprises audio content and the rendered media content comprises rendered audio content; and / or the media content comprises video content, and the rendered media content comprises rendered video content; This relates to the second device described in EEE35. EEE37 is The second apparatus according to EEE36, wherein the audio content is one of First Order Ambisonics (FOA) based, Higher Order Ambisonics (HOA) based, object based or channel based audio content or a combination of two or more of FOA based, HOA based, object based or channel based audio content. EEE38 is The second apparatus according to EEE36, wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content or channel-based audio content or a combination of two or more of binaural audio content, FOA audio content, HOA audio content or channel-based audio content. EEE39 is obtaining predicted attitude information based on the attitude information and previous attitude information; and adapted to render the media content based on the predicted pose information to obtain the rendered media content. The second device according to any one of claims EEE35 to EEE38. EEE40 is and adapted to transmit the predicted pose information to the first device together with the rendered media content. This relates to the second device described in EEE39. EEE41 is The second device according to any one of claims 39 to 40, wherein the predicted pose information is predicted for an estimate of a time at which the rendered media content is expected to be processed by the first device for reproduction. EEE42 is The second device according to any one of claims 35 to 41, wherein the rendered media content is transmitted to the first device in uncompressed form. EEE43 is, The second device according to any one of claims 35 to 42, further adapted to encode the rendered media content prior to transmission to the first device. EEE44 is The estimate of the time at which the rendered audio content is expected to be processed by the first device for reproduction includes an estimate of the time required to encode and decode the rendered audio content and / or an estimate of the time required to transmit the rendered media content to the first device. EEE45 is The second device according to any of the preceding claims, including EEE39 or EEE39, wherein the predicted pose information is obtained further based on an estimate of a time required to encode and decode the rendered media content and / or an estimate of a time required to transmit the rendered media content to the first device. EEE46 is determining gradient information indicative of how the rendered media content changes in response to changes in the pose information; and further adapted to transmit the gradient information to the first device together with the rendered media content. The second device according to any one of claims EEE35 to 45. EEE47 is the media content comprises audio content, and the rendered media content comprises rendered audio content; the second device is further adapted to receive environmental information from the first device indicative of an acoustic characteristic of an environment in which the first device is located; the rendering of the media content is further based on the environmental information. The second device according to any one of claims 35 to 46. EEE48 is, the media content comprises audio content, and the rendered media content comprises rendered audio content; the second device is further adapted to receive morphological information from the first device indicative of a morphology of a user or a part of a user; the rendering of the media content is further based on the morphological information; This relates to the second device described in EEE35. EEE49 is 1. A first device that reproduces media content provided by a second device, the first device comprising: acquiring attitude information indicative of a position and / or orientation of a user of the first device; transmitting the attitude information to the second device; receiving rendered media content from the second device, the rendered media content obtained by rendering the media content based on the pose information; and and reproducing the rendered media content. Regarding the first device. The EEE50 is the media content comprises audio content and the rendered media content comprises rendered audio content; and / or the media content comprises video content, and the rendered media content comprises rendered video content; This relates to the first device described in EEE49. EEE51 is the media content comprises audio content, and the rendered media content comprises rendered audio content; the first device is further adapted to generate an audible representation of the rendered audio content. This relates to the first device described in EEE49. EEE52 is The first device according to EEE50 or 51, wherein the audio content is one of First Order Ambisonics (FOA) based, Higher Order Ambisonics (HOA) based, object based or channel based audio content or a combination of two or more of FOA based, HOA based, object based or channel based audio content. EEE53 is The first device of any one of claims EEE50 to 52, wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content or channel-based audio content or a combination of two or more of binaural audio content, FOA audio content, HOA audio content or channel-based audio content. EEE54 is receiving from the second device pose information used to render the media content along with the rendered media content; comparing the pose information used to render the media content with the actual pose information; and further adapted to update the rendered media content based on a result of the comparison. The first device according to any one of claims EEE49 to 53. The EEE55 is The first device according to EEE54, wherein the actual pose information is pose information obtained at the time the rendered media content is processed for reproduction by the first device. EEE56 is obtaining predicted attitude information based on the attitude information and previous attitude information; and further adapted to transmit the predicted attitude information to the second device. The first device according to any one of claims EEE49 to 55. EEE57 is The first device according to claim 8, wherein the predicted pose information is predicted for an estimate of a time at which the rendered media content is expected to be processed by the first device for reproduction. EEE58 is The first device according to any one of claims EEE49 to 57, wherein the rendered media content is received from the second device in uncompressed form. EEE59 is The first device according to any one of claims 49 to 58, wherein the first device is further adapted to decode the encoded rendered media content. EEE60 is The estimate of the time at which the rendered audio content is expected to be processed by the first device for reproduction comprises an estimate of the time required to encode and decode the rendered audio content and / or an estimate of the time required to transmit the rendered media content to the first device. EEE61 is comparing pose information used to render the media content with current pose information; and further adapted to update the rendered media content based on a result of the comparison. The first device according to any one of claims EEE49 to 60. EEE62 is receiving gradient information from the second device along with the rendered media content, the gradient information indicating how the rendered media content changes in response to changes in the pose information; comparing pose information used to render the media content with current pose information; and further adapted to update the rendered media content based on the gradient information and a result of the comparison. The first device according to any one of claims EEE49 to 61. EEE63 is the media content comprises audio content, and the rendered media content comprises rendered audio content; The first device is further adapted to transmit environmental information indicative of an acoustic characteristic of an environment in which the first device is located to the second device; the rendering of the media content is further based on the environmental information. The first device according to any one of claims EE49 to EE62. EEE64 is the media content comprises audio content, and the rendered media content comprises rendered audio content; The first device is further adapted to transmit morphological information indicative of a morphology of a user or a part of a user to the second device; the rendering of the media content is further based on the morphological information; The first device according to any one of claims EE49 to EE63. EEE65 relates to a method for rendering audio content, the method comprising receiving, by a transmitter (S) device, user position and / or orientation data and transmitting corresponding pre-rendered content, typically derived from an object-5 based or HOA representation. EEE66 relates to the method as recited in EEE65, wherein the pre-rendered signal generated by the transmitter can be binaural, FOA / B format, HOA, or any type of channel-based rendering. EEE67 relates to the method according to EEE65 or 66, further comprising transmitting uncompressed pre-rendered content. EEE68 relates to the method of EEE65 or 66, further comprising encoding the pre-rendered content and transmitting the encoded pre-rendered content. EEE69 relates to the method of any one of EEE65 to 68, further comprising receiving the pre-rendered content by a receiver. EEE70 relates to the method of any one of EEE65 to 69, further comprising decoding, by a receiver, the pre-rendered, pre-encoded binauralized content. EEE71 relates to the method of any one of EEE65 to 70, wherein the user position and / or orientation data comprises a local pose indicating the user's position and orientation in world space. EEE72 relates to the method of any one of EEE65 to 71, wherein the user location data is transmitted from the receiver to the transmitter. EEE73 relates to the method of any one of EEE65 to 72, further comprising transmitting user position data used for the pre-rendered binauralized content back to the receiver. EEE74 relates to the method of any one of EEE65 to 73, further comprising extrapolating the pre-rendered content based on received user location data and local location data to determine updated content. EEE75 relates to the method of any one of EEE65 to 74, further comprising transmitting morphological data about the user (e.g. head size, head shape) for personalized binaural processing. EEE76 relates to the method of any one of EEE65 to 75, further comprising transmitting data on the BRIR function and / or the room characterization. EEE77 relates to the method of any one of EEE65 to 76, further comprising performing binaural rendering and personalization at the receiving side based on determining that the content is transmitted in a listener-transparent manner (e.g. without including HRTFs). EEE78 relates to the method of any one of EEE65 to 77, further comprising providing user position and / or orientation data P(t0) at time t1. EEE79 relates to the method of EEE67, wherein the uncompressed pre-rendered content is binauralized uncompressed pre-rendered content.

[0101] Several aspects will be described. [Aspect 1] 1. A method of processing media content for reproduction by a first device, comprising: obtaining attitude information indicative of a position and / or orientation of a user; transmitting the orientation information to a second device providing the media content; rendering the media content based on the pose information to obtain rendered media content; transmitting the rendered media content to the first device for reproduction. method. [Aspect 2] the media content comprises audio content and the rendered media content comprises rendered audio content; and / or the media content comprises video content, and the rendered media content comprises rendered video content; The method of embodiment 1. [Aspect 3] the media content comprises audio content, and the rendered media content comprises rendered audio content; The method further includes generating an audible representation of the rendered audio content. The method of embodiment 1. Aspect 4 The method of embodiment 2, wherein the audio content is one of First Order Ambisonics (FOA)-based, Higher Order Ambisonics (HOA)-based, object-based or channel-based audio content, or a combination of two or more of FOA-based, HOA-based, object-based or channel-based audio content. Aspect 5 3. The method of claim 2, wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content or channel-based audio content or a combination of two or more of binaural audio content, FOA audio content, HOA audio content or channel-based audio content. Aspect 6 The rendering is: obtaining predicted attitude information based on the attitude information and previous attitude information; and rendering the media content based on the predicted pose information to obtain the rendered media content. The method of embodiment 1. Aspect 7 transmitting the predicted pose information to the first device along with the rendered media content. The method of embodiment 6. Aspect 8 comparing the predicted attitude information with actual attitude information; and updating the rendered media content based on the results of the comparison. The method of embodiment 7. Aspect 9 the predicted pose information is predicted for an estimate of a time at which the rendered media content is expected to be processed by the first device for reproduction; the actual pose information being pose information obtained at the time the rendered media content is actually processed for reproduction by the first device; The method of embodiment 8. Aspect 10 2. The method of claim 1, wherein the rendered media content is transmitted to the first device in an uncompressed form. Aspect 11 encoding the rendered media content prior to transmission to the first device, and decoding the encoded rendered media content after receipt at the first device. The method of embodiment 1. Aspect 12 The method of claim 9, wherein the estimate of the time at which the rendered audio content is expected to be processed by the first device for reproduction includes an estimate of the time required to encode and decode the rendered audio content and / or an estimate of the time required to transmit the rendered media content to the first device. Aspect 13 The method of claim 6, wherein the predicted attitude information is obtained further based on an estimate of a time required to encode and decode the rendered media content and / or an estimate of a time required to transmit the rendered media content to the first device. Aspect 14 comparing pose information used to render the media content with current pose information; and updating the rendered media content based on the results of the comparison. The method of embodiment 1. Aspect 15 determining, at the second device, gradient information indicative of how the rendered media content changes in response to changes in the pose information; transmitting the gradient information along with the rendered media content to the first device; and comparing, at the first device, pose information used to render the media content with current pose information. and updating the rendered media content based on the gradient information and the comparison. The method of embodiment 1. Aspect 16 the media content comprises audio content, and the rendered media content comprises rendered audio content; The method further includes transmitting environmental information to the second device, the environmental information being indicative of an acoustic characteristic of an environment in which the first device is located; the rendering of the media content is further based on the environmental information. The method of embodiment 1. Aspect 17 the media content comprises audio content, and the rendered media content comprises rendered audio content; The method further includes transmitting morphological information indicative of a morphology of a user or a portion of a user to the second device; the rendering of the media content is further based on the morphological information; The method of embodiment 1. Aspect 18 1. A system having a first device for reproducing media content and a second device for storing said media content, comprising: The first device comprises: Obtaining attitude information indicating the user's position and / or orientation; · adapted to transmit the attitude information to the second device; The second device comprises: rendering the media content based on the pose information to obtain rendered media content; adapted to transmit the rendered media content to the first device for reproduction; system. Aspect 19 the media content comprises audio content and the rendered media content comprises rendered audio content; and / or the media content comprises video content, and the rendered media content comprises rendered video content; 20. The system of embodiment 18. Aspect 20 the media content comprises audio content, and the rendered media content comprises rendered audio content; the first device is further adapted to generate an audible representation of the rendered audio content. 20. The system of embodiment 18. Aspect 21 The system of embodiment 19, wherein the audio content is one of First Order Ambisonics (FOA)-based, Higher Order Ambisonics (HOA)-based, object-based or channel-based audio content, or a combination of two or more of FOA-based, HOA-based, object-based or channel-based audio content. Aspect 22 The system of aspect 19, wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content or channel-based audio content, or a combination of two or more of binaural audio content, FOA audio content, HOA audio content or channel-based audio content. Aspect 23 The second device further comprises: obtaining predicted attitude information based on the attitude information and previous attitude information; and adapted to render the media content based on the predicted pose information to obtain the rendered media content. 20. The system of embodiment 18. Aspect 24 the second device further comprising: and adapted to transmit the predicted pose information to the first device together with the rendered media content. 24. The system of claim 23. Aspect 25 the first device further comprising: Comparing the predicted attitude information with the actual attitude information; and further adapted to update the rendered media content based on a result of the comparison. 25. The system of embodiment 24. Aspect 26 the predicted pose information is predicted for an estimate of a time at which the rendered media content is expected to be processed by the first device for reproduction; the actual pose information being pose information obtained at the time the rendered media content is actually processed for reproduction by the first device; 26. The system of claim 25. Aspect 27 20. The system of embodiment 18, wherein the rendered media content is transmitted to the first device in an uncompressed form. Aspect 28 the second device is further adapted to encode the rendered media content prior to transmission to the first device; the first device is further adapted to decode the encoded rendered media content after receipt at the first device. 20. The system of embodiment 18. Aspect 29 The system of embodiment 26, wherein the estimate of the time at which the rendered audio content is expected to be processed by the first device for reproduction includes an estimate of the time required to encode and decode the rendered audio content and / or an estimate of the time required to transmit the rendered media content to the first device. Aspect 30 24. The system of claim 23, wherein the predicted attitude information is obtained further based on an estimate of a time required to encode and decode the rendered media content and / or an estimate of a time required to transmit the rendered media content to the first device. Aspect 31 the first device further comprising: comparing pose information used to render the media content with current pose information; and adapted to update the rendered media content based on a result of the comparison. 20. The system of embodiment 18. Aspect 32 the second device further comprising: determining gradient information indicative of how the rendered media content changes in response to changes in the pose information; adapted to transmit the gradient information to the first device together with the rendered media content; the first device further comprising: comparing pose information used to render the media content with current pose information; and updating the rendered media content based on the gradient information and a result of the comparison. 20. The system of embodiment 18. Aspect 33 the media content comprises audio content, and the rendered media content comprises rendered audio content; The first device is further adapted to transmit environmental information indicative of an acoustic characteristic of an environment in which the first device is located to the second device; the rendering of the media content is further based on the environmental information. 20. The system of embodiment 18. Aspect 34 the media content comprises audio content, and the rendered media content comprises rendered audio content; The first device is further adapted to transmit morphological information indicative of a morphology of a user or a part of a user to the second device; the rendering of the media content is further based on the morphological information; 20. The system of embodiment 18. Aspect 35 A second device providing media content for reproduction by the first device, the second device comprising: receiving attitude information indicative of a position and / or orientation of a user of the first device; rendering the media content based on the pose information to obtain rendered media content; adapted to transmit the rendered media content to the first device for reproduction. The second device. Aspect 36 the media content comprises audio content and the rendered media content comprises rendered audio content; and / or the media content comprises video content, and the rendered media content comprises rendered video content; 36. The second device of embodiment 35. Aspect 37 The second apparatus of embodiment 36, wherein the audio content is one of First Order Ambisonics (FOA)-based, Higher Order Ambisonics (HOA)-based, object-based or channel-based audio content, or a combination of two or more of FOA-based, HOA-based, object-based or channel-based audio content. Aspect 38 The second apparatus of embodiment 36, wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content or channel-based audio content, or a combination of two or more of binaural audio content, FOA audio content, HOA audio content or channel-based audio content. Aspect 39 obtaining predicted attitude information based on the attitude information and previous attitude information; and adapted to render the media content based on the predicted pose information to obtain the rendered media content. 36. The second device of embodiment 35. Aspect 40 and adapted to transmit the predicted pose information to the first device together with the rendered media content. 40. The second device of embodiment 39. Aspect 41 40. The second device of embodiment 39, wherein the predicted pose information is predicted for an estimate of a time at which the rendered media content is expected to be processed by the first device for reproduction. Aspect 42 36. The second device of embodiment 35, wherein the rendered media content is transmitted to the first device in an uncompressed form. Aspect 43 36. The second device of embodiment 35, further adapted to encode the rendered media content prior to transmission to the first device. Aspect 44 The second device of embodiment 41, wherein the estimate of the time at which the rendered audio content is expected to be processed by the first device for reproduction includes an estimate of the time required to encode and decode the rendered audio content and / or an estimate of the time required to transmit the rendered media content to the first device. Aspect 45 The second device of embodiment 39, wherein the predicted attitude information is obtained further based on an estimate of the time required to encode and decode the rendered media content and / or an estimate of the time required to transmit the rendered media content to the first device. Aspect 46 determining gradient information indicative of how the rendered media content changes in response to changes in the pose information; and further adapted to transmit the gradient information to the first device together with the rendered media content. 36. The second device of embodiment 35. Aspect 47 the media content comprises audio content, and the rendered media content comprises rendered audio content; the second device is further adapted to receive environmental information from the first device indicative of an acoustic characteristic of an environment in which the first device is located; the rendering of the media content is further based on the environmental information. 36. The second device of embodiment 35. Aspect 48 the media content comprises audio content, and the rendered media content comprises rendered audio content; the second device is further adapted to receive morphological information from the first device indicative of a morphology of a user or a part of a user; the rendering of the media content is further based on the morphological information; 36. The second device of embodiment 35. Aspect 49 1. A first device that reproduces media content provided by a second device, the first device comprising: acquiring attitude information indicative of a position and / or orientation of a user of the first device; transmitting the attitude information to the second device; receiving rendered media content from the second device, the rendered media content obtained by rendering the media content based on the pose information; and and reproducing the rendered media content. First device. Aspect 50 the media content comprises audio content and the rendered media content comprises rendered audio content; and / or the media content comprises video content, and the rendered media content comprises rendered video content; 50. The first device of embodiment 49. Aspect 51 the media content comprises audio content, and the rendered media content comprises rendered audio content; the first device is further adapted to generate an audible representation of the rendered audio content. 50. The first device of embodiment 49. Aspect 52 The first apparatus of embodiment 50, wherein the audio content is one of First Order Ambisonics (FOA)-based, Higher Order Ambisonics (HOA)-based, object-based or channel-based audio content, or a combination of two or more of FOA-based, HOA-based, object-based or channel-based audio content. Aspect 53 The first device of aspect 50, wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content or channel-based audio content, or a combination of two or more of binaural audio content, FOA audio content, HOA audio content or channel-based audio content. Aspect 54 receiving from the second device pose information used to render the media content along with the rendered media content; comparing the pose information used to render the media content with the actual pose information; and further adapted to update the rendered media content based on a result of the comparison. 50. The first device of embodiment 49. Aspect 55 55. The first device of embodiment 54, wherein the actual pose information is pose information obtained at the time the rendered media content is processed for reproduction by the first device. Aspect 56 obtaining predicted attitude information based on the attitude information and previous attitude information; and further adapted to transmit the predicted attitude information to the second device. 50. The first device of embodiment 49. Aspect 57 57. The first device of embodiment 56, wherein the predicted pose information is predicted for an estimate of a time at which the rendered media content is expected to be processed by the first device for reproduction. Aspect 58 50. The first device of embodiment 49, wherein the rendered media content is received from the second device in an uncompressed form. Aspect 59 50. The first device of embodiment 49, wherein the first device is further adapted to decode the encoded rendered media content. Aspect 60 A first device as described in embodiment 57, wherein the estimate of the time at which the rendered audio content is expected to be processed by the first device for reproduction includes an estimate of the time required to encode and decode the rendered audio content and / or an estimate of the time required to transmit the rendered media content to the first device. Aspect 61 comparing pose information used to render the media content with current pose information; and further adapted to update the rendered media content based on a result of the comparison. 50. The first device of embodiment 49. Aspect 62 receiving gradient information from the second device along with the rendered media content, the gradient information indicating how the rendered media content changes in response to changes in the pose information; comparing pose information used to render the media content with current pose information; and further adapted to update the rendered media content based on the gradient information and a result of the comparison. 50. The first device of embodiment 49. Aspect 63 the media content comprises audio content, and the rendered media content comprises rendered audio content; The first device is further adapted to transmit environmental information indicative of an acoustic characteristic of an environment in which the first device is located to the second device; the rendering of the media content is further based on the environmental information. 50. The first device of embodiment 49. Aspect 64 the media content comprises audio content, and the rendered media content comprises rendered audio content; The first device is further adapted to transmit morphological information indicative of a morphology of a user or a part of a user to the second device; the rendering of the media content is further based on the morphological information; 50. The first device of embodiment 49.

Claims

1. 1. A system having a first device for reproducing media content and a second device for storing said media content, comprising: The first device comprises: - obtaining head pose information indicative of the user's position and / or orientation; - adapted to transmit the head pose information to the second device; The second device comprises: - rendering the media content based on the head pose information to obtain rendered media content; - adapted to transmit the rendered media content to the first device for reproduction; The second device further comprises: obtaining predicted head pose information based on the head pose information and previous head pose information; adapted to render the media content based on the predicted head pose information to obtain the rendered media content. system.

2. the media content comprises audio content and the rendered media content comprises rendered audio content; and / or the media content comprises video content, and the rendered media content comprises rendered video content; The system of claim 1 .

3. the media content comprises audio content, and the rendered media content comprises rendered audio content; the first device is further adapted to generate an audible representation of the rendered audio content. The system of claim 1 .

4. 3. The system of claim 2, wherein the audio content is one of First Order Ambisonics (FOA)-based, Higher Order Ambisonics (HOA)-based, object-based or channel-based audio content, or a combination of two or more of FOA-based, HOA-based, object-based or channel-based audio content.

5. 3. The system of claim 2, wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content, or channel-based audio content, or a combination of two or more of binaural audio content, FOA audio content, HOA audio content, or channel-based audio content.

6. The second device further comprising: adapted to transmit the predicted head pose information together with the rendered media content to the first device. The system of claim 1 .

7. The first device further comprising: comparing the predicted head pose information with actual head pose information; and further adapted to update the rendered media content based on a result of the comparison. The system of claim 6.

8. the predicted head pose information is predicted for an estimate of a time at which the rendered media content is expected to be processed by the first device for reproduction; the actual head pose information being head pose information obtained at the time the rendered media content is actually processed for reproduction by the first device. The system of claim 7.

9. The system of claim 1 , wherein the rendered media content is transmitted to the first device in an uncompressed form.

10. the second device is further adapted to encode the rendered media content prior to transmission to the first device; the first device is further adapted to decode the encoded rendered media content after receipt at the first device. The system of claim 1 .

11. 10. The system of claim 8, wherein the estimate of the time at which the rendered audio content is expected to be processed by the first device for reproduction includes an estimate of the time required to encode and decode the rendered audio content and / or an estimate of the time required to transmit the rendered media content to the first device.

12. 6. The system of claim 5, wherein the predicted head pose information is obtained further based on an estimate of a time required to encode and decode the rendered media content and / or an estimate of a time required to transmit the rendered media content to the first device.

13. The first device further comprising: comparing head pose information used to render the media content with current head pose information; and adapted to update the rendered media content based on a result of the comparison. The system of claim 1 .

14. 1. A system having a first device for reproducing media content and a second device for storing said media content, comprising: The first device comprises: - Obtaining attitude information indicating the user's position and / or orientation; - adapted to transmit the attitude information to the second device; The second device comprises: - rendering the media content based on the pose information to obtain rendered media content; - adapted to transmit the rendered media content to the first device for reproduction; The second device further comprising: determining gradient information indicative of how the rendered media content changes in response to changes in the pose information; adapted to transmit the gradient information to the first device together with the rendered media content; The first device further comprising: comparing pose information used to render the media content with current pose information; and adapted to update the rendered media content based on the gradient information and a result of the comparison. system.

15. A second device providing media content for reproduction by the first device, the second device comprising: receiving head pose information indicative of a position and / or orientation of a user of the first device; rendering the media content based on the head pose information to obtain rendered media content; adapted to transmit the rendered media content to the first device for reproduction; The second device further comprises: obtaining predicted head pose information based on the head pose information and previous head pose information; adapted to render the media content based on the predicted head pose information to obtain the rendered media content. The second device.

16. the media content comprises audio content and the rendered media content comprises rendered audio content; and / or the media content comprises video content, and the rendered media content comprises rendered video content; 16. A second device according to claim 15.

17. 17. The second device of claim 16, wherein the audio content is one of First Order Ambisonics (FOA)-based, Higher Order Ambisonics (HOA)-based, object-based or channel-based audio content, or a combination of two or more of FOA-based, HOA-based, object-based or channel-based audio content.

18. 17. The second device of claim 16, wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content or channel-based audio content, or a combination of two or more of binaural audio content, FOA audio content, HOA audio content or channel-based audio content.

19. adapted to transmit the predicted head pose information together with the rendered media content to the first device.

16. A second device according to claim 15.

20. 16. The second device of claim 15, wherein the predicted head pose information is predicted for an estimate of a time at which the rendered media content is expected to be processed by the first device for reproduction.

21. The second device of claim 15 , wherein the rendered media content is transmitted to the first device in an uncompressed form.

22. The second device of claim 15 , further adapted to encode the rendered media content prior to transmission to the first device.

23. 21. The second device of claim 20, wherein the estimate of the time at which the rendered audio content is expected to be processed by the first device for reproduction includes an estimate of the time required to encode and decode the rendered audio content and / or an estimate of the time required to transmit the rendered media content to the first device.

24. 16. The second device of claim 15, wherein the predicted head pose information is obtained further based on an estimate of a time required to encode and decode the rendered media content and / or an estimate of a time required to transmit the rendered media content to the first device.

25. A second device providing media content for reproduction by the first device, the second device comprising: receiving attitude information indicative of a position and / or orientation of a user of the first device; rendering the media content based on the pose information to obtain rendered media content; adapted to transmit the rendered media content to the first device for reproduction; The second device comprises: determining gradient information indicative of how the rendered media content changes in response to changes in the pose information; and further adapted to transmit the gradient information to the first device together with the rendered media content. The second device.

Images