Optimizing Audio Delivery for Virtual Reality Applications

Abstract

To provide techniques, systems, methods, and instructions for enabling a virtual reality VR, an augmented reality AR, a mixed reality MR, or a 360-degree video environment with low complexity.SOLUTION: A system 102 comprises at least one media video decoder for decoding video signals from video streams 1800 and at least one audio decoder 104 for decoding audio signals 108 from at least one audio stream 106. The system requests at least one audio stream 106 and / or one audio element of an audio stream 106 and / or one adaptation set to a server 120 on the basis of at least a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual positional data 110 provided as a feedback from a media consumption device 180.SELECTED DRAWING: Figure 1.8

Description

[Technical field]

[0001] [Background technology]

[0002] In a Virtual Reality (VR) environment, or similarly Augmented Reality (AR) or Mixed Reality (MR) or 360° video environment, the user can typically visualize the entire 360° content using, for example, a Head Mounted Display (HMD) and listen to it through headphones (or equivalently through speakers with the correct rendering depending on their position).

[0003] In a simple use case, content is created such that only one audio / video scene (e.g. a 360 degree video) is playing at any given moment. The audio / video scene has a fixed position (e.g. a sphere with the user located at the center) and the user cannot move within the scene, they can only rotate their head in different directions (yaw, pitch, roll). In this case, different video and audio are played (different viewports are displayed) based on the orientation of the user's head.

[0004] For video, the video content is delivered for the entire 360-degree scene along with metadata to describe the rendering process (e.g., stitching information, projection mapping, etc.) and selected based on the current user viewport, while for audio, the content is the same for the entire scene. Based on the metadata, the audio content is adapted to the current user viewport (e.g., audio objects are rendered differently based on viewport / user orientation information). Note that 360-degree content refers to any type of content consisting of multiple viewing angles simultaneously that can be selected by the user (e.g., by the user's head orientation or a remote control device).

[0005] In more complex scenarios, the audio content may also change as the user moves in the VR scene or "jumps" from one scene to the next (e.g. an audio source that is not audible in one scene becomes audible in the next - "door opens"). Existing systems encode complete audio scenes into one stream and may encode them into additional streams as needed (depending on the main stream). Such systems are known as next generation audio systems (e.g. MPEG-H 3D Audio, etc.). Such use cases may include:

[0006] Example 1: The user chooses to enter a new room and the entire audio / video scene changes Example 2: When a user moves through a VR scene and opens a door and walks through it, this means there needs to be an audio transition from one scene to the next. For the purpose of describing this scenario, the concept of discrete viewpoints in space is introduced, as discrete locations in a space (or VR environment) at which different audio / video content is available.

[0007] A "straightforward" solution would be to have a real-time encoder that changes the encoding (number of audio elements, spatial information, etc.) based on feedback from the playback device about the user's position / orientation. This solution would imply, for example, very complex communication between client and server in a streaming environment.

[0008] Clients (which are typically assumed to use only simple logic) need sophisticated mechanisms to communicate not only requests for the various streams, but also complex information about the encoding details that allow appropriate content to be served based on the user's location.

[0009] · A media server is usually pre-populated with the various streams (formatted in a specific way that allows "segment by segment" delivery) and the main function of the server is to provide information about the available streams and deliver them when requested. To enable scenarios that allow encoding based on feedback from playback devices, the media server needs advanced communication links with multiple live media encoders, and the ability to create on the fly all the signaling information that can change in real time (e.g. Media Presentation Descriptions).

[0010] While such systems can be envisioned, their complexity and computational requirements are beyond the capabilities and features of the instruments and systems currently available or that may be developed in the coming decades.

[0011] Alternatively, content representing the complete VR environment (the "complete world") could be delivered all the time, which would solve the problem but would require huge bitrates that would exceed the capacity of the available communication links.

[0012] This is complex in a real-time environment and in order to enable such use cases using available systems, alternative solutions have been proposed that allow this functionality with low complexity.

[0013] 2. Terms and Definitions The following terms are used in the art:

[0014] Audio Elements: Audio signals that can be represented for example as audio objects, audio channels, scene-based audio (Higher Order Ambisonics - HOA), or any combination of all.

[0015] Region of Interest (ROI): A single area of ​​video content (or a displayed or simulated environment) that is of interest to the user at a given point in time. This is usually, for example, a region on a sphere, or a selection of a polygon from a 2D map. The ROI identifies a specific area for a specific purpose and defines the boundaries of the object under consideration.

[0016] User location information: location information (e.g. x, y, z coordinates), orientation information (yaw, pitch, roll), movement direction, movement speed, etc.

[0017] Viewport: The part of the spherical video that is currently displayed and being viewed by the user.

[0018] Viewpoint: The center point of the viewport.

[0019] 360-degree video (also known as immersive video or spherical video): In the context of this document, it describes video content that contains multiple views (viewports) in one direction at the same time. Such content can be created, for example, using an omnidirectional camera or collection of cameras. During playback, the viewer can control the viewing direction.

[0020] · Media Presentation Description (MPD) is a syntax, e.g. XML, that contains information about media segments, their relationships and the information needed to select them.

[0021] An adaptation set contains a media stream or a set of media streams. In the simplest case there is one adaptation set containing all the audio and video of the content, but to reduce bandwidth each stream can be split into different adaptation sets. A common case is to have one video adaptation set and several audio adaptation sets (one for each supported language). An adaptation set can also contain subtitles or any metadata.

[0022] Representations allow an adaptation set to contain the same content encoded in different ways. In most cases, representations are offered at multiple bitrates, allowing clients to request the highest quality content they can play without waiting for buffering. Representations can also be encoded in different codecs, thus supporting clients with a variety of supported codecs.

[0023] In the context of this application, the notion of adaptation set is more generally used and may actually refer to a representation. Also, media streams (audio / video streams) are typically encapsulated into media segments, which are the actual media files that are first played by a client (e.g., a DASH client). Media segments can use different formats, such as the ISO Base Media File Format (ISOBMFF), which is similar to the MPEG-4 container format, or MPEG-2 Transport Stream (TS). The encapsulation into media segments and in the different representations / adaptation sets is independent of the method described here, which applies to all the different options.

[0024] Additionally, although the description of the method in this document focuses on DASH server and client communication, the method is general enough to work with other delivery environments such as MMT, MPEG-2 TS, DASH-ROUTE, and file formats for file playback.

[0025] In general, an adaptation set is layered above a stream and can contain metadata (e.g., associated with position). A stream can contain multiple audio elements. An audio scene can be associated with multiple streams that are delivered as part of multiple adaptation sets.

[0026] 3. Current Solution The current solution is as follows:

[0027] [1].ISO / IEC 23008-3:2015,Information technology--High efficiency coding and media delivery in heterogeneous environments--Part 3:3D audi

[0028] [2].N16950,Study of ISO / IEC DIS 23000-20 Omnidirectional Media Format Current solutions are limited in that they can provide an isolated VR experience in one fixed location, allowing the user to orient but not move within the VR environment. [Prior art documents] [Non-patent literature]

[0029] [Non-Patent Document 1] ISO / IEC 23008-3:2015,Information technology--High efficiency coding and media delivery in heterogeneous environments--Part 3:3D audio [Non-Patent Document 2] N16950,Study of ISO / IEC DIS 23000-20 Omnidirectional Media Format Summary of the Invention

[0030] According to one embodiment, a system for a virtual reality VR, augmented reality AR, mixed reality MR, or 360 degree video environment may be configured to receive a video stream and an audio stream to be played on a media consumption device, the system may include at least one media video decoder configured to decode a video signal from the video stream to represent a VR, AR, MR, or 360 degree video environment scene to a user, and at least one audio decoder configured to decode an audio signal from the at least one audio stream, and the system may be configured to request from a server at least one audio stream and / or one audio element and / or one adaptation set of the audio stream based at least on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data.

[0031] According to one embodiment, the system may be configured to provide a server with a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data in order to retrieve at least one audio stream and / or one audio element of an audio stream and / or one adaptation set from the server.

[0032] In one embodiment, at least one scene may be associated with at least one audio element, each audio element associated with a position and / or area in the visual environment where the audio element is audible, and different audio streams may be provided for different user positions and / or viewports and / or head orientations and / or movement data and / or interaction metadata and / or virtual position data within the scene. According to another aspect, the system may be configured to determine whether to play at least one audio element and / or one adaptation set of the audio stream relative to a current user's viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position in the scene, and the system may be configured to request and / or receive the at least one audio element at the current user's virtual position.

[0033] According to one aspect, the system may be configured to predictively determine whether at least one audio element and / or one adaptation set of an audio stream will be relevant and / or audible based on at least a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, the system may be configured to request and / or receive the at least one audio element and / or audio stream and / or adaptation set at a particular user virtual position prior to a predicted user movement and / or interaction in a scene, and the system may be configured, upon receipt, to play the at least one audio element and / or audio stream at the particular user virtual position after the user movement and / or interaction in the scene.

[0034] One embodiment of the system may be configured to request and / or receive at least one audio element at a lower bitrate and / or quality level at a user's virtual position before the user's movement and / or interaction in the scene, and the system may be configured to request and / or receive at least one audio element at a higher bitrate and / or quality level at a user's virtual position after the user's movement and / or interaction in the scene.

[0035] According to one aspect, the system may be configured such that at least one audio element is associated with at least one scene, each audio element being associated with a location and / or area within the visual environment associated with the scene, and the system may be configured to request and / or receive streams at a higher bitrate and / or quality for audio elements closer to the user than for audio elements farther from the user.

[0036] According to one aspect of the system, at least one audio element may be associated with at least one scene, and at least one audio element may be associated with a location and / or area within the visual environment associated with the scene, and the system may be configured to request different streams at different bitrates and / or quality levels for the audio elements based on a relevance and / or auditability level at each user's virtual location in the scene, and the system may be configured to request audio streams at higher bitrates and / or quality levels for audio elements that are more relevant and / or more audible at the current user's virtual location, and / or to request audio streams at lower bitrates and / or quality levels for audio elements that are less relevant and / or less audible at the current user's virtual location.

[0037] In one embodiment of the system, at least one audio element may be associated with a scene, each audio element associated with a position and / or area in a visual environment associated with the scene, and the system may be configured to periodically transmit the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data to a server, such that at a first location a higher bitrate and / or quality stream is provided from the server and at a second location a lower bitrate and / or quality stream is provided from the server, the first location being closer to the at least one audio element than the second location.

[0038] In one embodiment, the system may be defined for multiple visual environments, such as an environment in which multiple scenes are adjacent and / or proximate, where a first stream associated with a first current scene is provided and, when the user transitions to a second further scene, both a stream associated with the first scene and a second stream associated with the second scene are provided.

[0039] In one embodiment, the system may define multiple scenes for a first and a second visual environment, the first and second environments being adjacent and / or proximate environments, a first stream associated with the first scene being provided from the server for playback of the first scene when the user's position or virtual location is in the first environment associated with the first scene, a second stream associated with the second scene being provided from the server for playback of the second scene when the user's position or virtual location is in the second environment associated with the second scene, and both the first stream associated with the first scene and the second stream associated with the second scene being provided when the user's position or virtual location is in a transition position between the first scene and the second scene.

[0040] In one embodiment, the system may define a plurality of scenes for a first and a second visual environment, the first and second visual environments being adjacent and / or proximate environments, the system may be configured to request and / or receive a first stream associated with a first scene associated with the first environment for playback of the first scene when the user's virtual location is in the first environment, the system may be configured to request and / or receive a second stream associated with a second scene associated with the second environment for playback of the second scene when the user's virtual location is in the second environment, and the system may be configured to request and / or receive both the first stream associated with the first scene and the second stream associated with the second scene when the user's virtual location is in a transitional position between the first and second environments.

[0041] According to one aspect, the system may be configured such that a first stream associated with a first scene is acquired at a higher bitrate and / or quality when a user is in a first environment associated with the first scene, while a second stream associated with a second scene associated with a second environment is acquired at a lower bitrate and / or quality when the user is at the beginning of a transition position from the first scene to the second scene, and the first stream associated with the first scene is acquired at the lower bitrate and / or quality and the second stream associated with the second scene is acquired at a higher bitrate and / or quality, the lower bitrate and / or quality being lower than the higher bitrate and / or quality, when the user is at the end of the transition position from the first scene to the second scene.

[0042] According to one aspect, the system may be configured such that a plurality of scenes are defined for a plurality of environments such as adjacent and / or neighboring environments. The system may obtain a stream associated with a first current scene associated with a first current environment. If the distance of the user's position or virtual position from the scene boundary is less than a predetermined threshold, the system may further obtain an audio stream associated with a second adjacent and / or neighboring environment associated with a second scene.

[0043] According to one aspect, the system may be configured such that a plurality of scenes may be defined for a plurality of visual environments. The system requests and / or obtains a stream associated with the current scene at a higher bitrate and / or quality, and a stream associated with a second scene at a lower bitrate and / or quality, where the lower bitrate and / or quality is lower than the higher bitrate and / or quality.

[0044] According to one aspect, the system may be configured such that a plurality of N audio elements may be defined. When the distance of the user to the position or region of these audio elements is greater than a predetermined threshold, the N audio elements are processed to obtain a smaller number M (M < N) of audio elements associated with a position or region close to the position or region of the N audio elements. Thereby, when the distance of the user to the position or region of the N audio elements is less than a predetermined threshold, at least one audio stream associated with the N audio elements is provided to the system, or when the distance of the user to the position or region of the N audio elements is greater than a predetermined threshold, at least one audio stream associated with the M audio elements is provided to the system.

[0045] According to one aspect, the system may be configured such that at least one visual environment scene is associated with at least one plurality of N audio elements (N>=2), each audio element being associated with a position and / or region within the visual environment. At least one plurality of N audio elements may be provided in at least one representation at a high bitrate and / or quality level, and at least one plurality of N audio elements may be provided in at least one representation at a low bitrate and / or quality level. At least one representation is obtained by processing the N audio elements to obtain a smaller number M (M<N) of audio elements associated with a position or region close to the position or region of the N audio elements. The system may be configured to request a representation at a higher bitrate and / or quality level for an audio element when the audio element is more relevant and / or more audible at the current virtual position of the user in the scene, and the system may be configured to request a representation at a lower bitrate and / or quality level for an audio element when the audio element is less relevant and / or less audible at the current virtual position of the user in the scene.

[0046] According to one aspect, the system may be configured such that different streams are obtained for different audio elements when the user's distance and / or relevance and / or audible level and / or angular orientation is below a predetermined threshold.

[0047] In one embodiment, the system may be configured to request and / or obtain a stream based on the orientation of the user in the scene and / or the direction of the user's movement and / or the user's interaction.

[0048] In one embodiment, the viewport of the system may be associated with position and / or virtual position and / or movement data and / or the head.

[0049] According to one aspect, the system may be configured to provide different audio elements at different viewports, and the system may be configured to request and / or receive, when one first audio element is within a viewport, a first audio element at a higher bitrate than a second audio element that is not within the viewport.

[0050] According to one aspect, the system may be configured to request and / or receive a first audio stream and a second audio stream, where a first audio element of the first audio stream is more relevant and / or more audible than a second audio element of the second audio stream, and where the first audio stream is requested and / or received at a higher bitrate and / or quality than the bitrate and / or quality of the second audio stream.

[0051] According to one aspect, the system may be configured such that at least two visual environment scenes are defined, at least one first and second audio element is associated with a first scene associated with the first visual environment and at least one third audio element is associated with a second scene associated with the second visual environment, the system may be configured to obtain metadata describing the at least one second audio element further associated with the second visual environment scene, the system may be configured to request and / or receive at least the first and second audio elements when a user's virtual location is in the first visual environment, the system may be configured to request and / or receive at least the second and third audio elements when a user's virtual location is in the second visual environment scene, and the system may be configured to request and / or receive at least the first and second and third audio elements when a user's virtual location transitions between the first visual environment scene and the second visual environment scene.

[0052] An embodiment of the system may be configured such that at least one first audio element is provided in at least one audio stream and / or adaptation set, at least one second audio element is provided in at least one second audio stream and / or adaptation set, and at least one third audio element is provided in at least one third audio stream and / or adaptation set, at least a first visual environment scene is described by the metadata as a complete scene requiring at least the first and second audio streams and / or adaptation sets, and the second visual environment scene is described by the metadata as an incomplete scene requiring at least a third audio stream and / or adaptation set, as well as at least a second audio stream and / or adaptation set associated with the at least first visual environment scene, and the system includes a metadata processor configured to manipulate the metadata to enable merging of the second audio stream belonging to the first visual environment and the third audio stream associated with the second visual environment into a new single stream when the user's virtual location is in the second visual environment.

[0053] According to one aspect, the system includes a metadata processor configured to manipulate metadata in at least one audio stream prior to the at least one audio decoder based on a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data.

[0054] According to one aspect, the metadata processor may be configured to enable and / or disable at least one audio element in the at least one audio stream before the at least one audio decoder based on a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data; if the system determines that an audio element is not played anymore as a result of the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, the metadata processor may be configured to disable at least one audio element in the at least one audio stream before the at least one audio decoder; if the system determines that an audio element is played back as a result of the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, the metadata processor may be configured to enable at least one audio element in the at least one audio stream before the at least one audio decoder.

[0055] According to one aspect, the system may be configured to disable decoding of selected audio elements based on the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position.

[0056] According to one aspect, the system may be configured to merge at least one first audio stream associated with a current audio scene with at least one stream associated with an adjacent, nearby, and / or future audio scene.

[0057] According to one embodiment, the system may be configured to obtain and / or collect statistical or aggregated data regarding a user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data and send a request to a server associated with the statistical or aggregated data.

[0058] According to one aspect, the system may be configured to deactivate decoding and / or playback of at least one stream based on metadata associated with the at least one stream and based on a user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data.

[0059] According to one aspect, the system may be configured to manipulate metadata associated with a group of selected audio streams based on at least the user's current or estimated viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data to enable selection and / or enabling and / or activating audio elements that make up the audio scene to be played back and / or merging all selected audio streams into a single audio stream.

[0060] According to one aspect, the system may be configured to control requests for at least one stream from the server based on the distance of the user's location from boundaries of adjacent and / or proximate environments associated with different scenes, or other metrics associated with the user's location in the current environment or a prediction of the user's location in a future environment.

[0061] According to one aspect of the system, for each audio element or audio object, information may be provided from the server system, the information including descriptive information about the sound scene or where the audio element is active.

[0062] According to one aspect, the system may be configured to select between playing one scene and compositing, mixing, multiplexing, superimposing or combining at least two scenes, the two scenes being associated with different adjacent and / or nearby environments, based on current or future or viewport and / or head orientation and / or movement data and / or metadata and / or virtual position and / or user selection.

[0063] According to one aspect, the system may be configured to create or use at least adaptation sets, where several adaptation sets are associated with one audio scene, and / or additional information is provided associating each adaptation set with one viewpoint or one audio scene, and / or additional information is provided which may include information regarding boundaries of one audio scene, and / or information regarding the relationship between one adaptation set and one audio scene (e.g., an audio scene is encoded into three streams encapsulated in three adaptation sets), and / or information regarding connections between audio scene boundaries and multiple adaptation sets.

[0064] According to one aspect, the system may be configured to receive a stream of a scene associated with an adjacent or nearby environment and, upon detection of a boundary transition between the two environments, initiate decoding and / or playback of the stream of the adjacent or nearby environment.

[0065] According to one aspect, the system may be configured to operate as a client and a server configured to deliver video and / or audio streams to be played on a media consumption device.

[0066] According to one aspect, the system may be configured to enable requesting and / or receiving at least one first adaptation set including at least one audio stream associated with at least one first audio scene, requesting and / or receiving at least one second adaptation set including at least one second audio stream associated with at least two audio scenes including the at least one first audio scene, and merging the at least one first audio stream and the at least one second audio stream into a new audio stream to be decoded based on available metadata regarding a user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data, and / or information describing association of the at least one first adaptation set to the at least one first audio scene and / or association of the at least one second adaptation set to the at least one first audio scene.

[0067] According to one aspect, the system may be configured to receive information regarding a user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data and / or information characterizing changes triggered by a user's actions, and to receive information regarding the availability of adaptation sets and information describing an association of at least one adaptation set to at least one scene and / or viewpoint and / or viewport and / or position and / or virtual position and / or movement data and / or orientation.

[0068] According to one aspect, the system may be configured to determine whether to play at least one audio element from at least one audio scene embedded in the at least one stream and at least one additional audio element from at least one additional audio scene embedded in the at least one additional stream, and in case of a positive determination, perform an operation of merging or synthesizing or multiplexing or superimposing or combining the at least one additional stream of the additional audio scene into the at least one stream of the at least one audio scene.

[0069] According to one aspect, the system may be configured to manipulate audio metadata associated with the selected audio streams based on at least the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data to select and / or enable and / or activate audio elements that constitute the audio scene determined to be played, and to enable merging of all selected audio streams into a single audio stream.

[0070] According to one aspect, a server may be provided for delivering audio and video streams for a virtual reality VR, augmented reality AR, mixed reality MR, or 360 degree video environment to a client, where the video and audio streams are played on a media consumption device, and the server may include an encoder for encoding and / or a storage device for storing a video stream describing a visual environment, where the visual environment is associated with an audio scene, and the server may further include an encoder for encoding and / or a storage device for storing a plurality of streams and / or audio elements and / or adaptation sets to be delivered to the client, where the streams and / or audio elements and / or adaptation sets are associated with at least one audio scene, and the server is configured to select and deliver a video stream based on a request from the client, where the video stream is associated with the environment, and select an audio stream and / or audio element and / or adaptation set based on a request from the client, where the request is associated with at least a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, as well as an audio scene associated with the environment, and deliver the audio stream to the client.

[0071] According to one aspect, the streams may be encapsulated into adaptation sets, each adaptation set including multiple streams associated with different representations at different bit rates and / or qualities of the same audio content, and the selected adaptation set is selected based on a request from a client.

[0072] According to one aspect, the system may act as a client and a server.

[0073] According to one aspect, a system may include a server.

[0074] According to one aspect, a method may be provided for a virtual reality VR, augmented reality AR, mixed reality MR, or 360 degree video environment configured to receive video and audio streams to be played on a media consumption device (e.g., a playback device), comprising: decoding a video signal from the video stream for representation of the VR, AR, MR, or 360 degree video environment scene to a user; decoding an audio signal from the audio stream; and requesting and / or obtaining at least one audio stream from a server based on the user's current viewport and / or position data and / or head orientation and / or movement data and / or metadata and / or virtual position data and / or metadata.

[0075] According to one aspect, a computer program may be provided comprising instructions which, when executed by a processor, cause the processor to perform the method set out above. [Brief description of the drawings]

[0076] [Figure 1.1] FIG. 1 shows an example of the invention. [Figure 1.2] FIG. 1 shows an example of the invention. [Figure 1.3] FIG. 1 shows an example of the invention. [Figure 1.4] FIG. 1 shows an example of the invention. [Figure 1.5] FIG. 1 shows an example of the invention. [Figure 1.6] FIG. 1 shows an example of the invention. [Figure 1.7] FIG. 1 shows an example of the invention. [Figure 1.8] FIG. 1 shows an example of the invention. [Diagram 2] FIG. 1 illustrates an invention scenario. [Diagram 3] FIG. 1 illustrates an invention scenario. [Figure 4] FIG. 1 illustrates an invention scenario. [Diagram 5]FIG. 1 illustrates an invention scenario. [Figure 6] FIG. 1 illustrates an invention scenario. [Figure 7A] FIG. 1 illustrates the method of the invention. [Figure 7B] FIG. 1 illustrates the method of the invention. [Figure 8A] FIG. 1 illustrates the method of the invention. [Figure 8B] FIG. 1 illustrates the method of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0077] Hereinafter (eg, starting with FIG. 1.1), examples of systems according to aspects of the present invention are disclosed.

[0078] An example system of the present invention (which may be embodied by different examples disclosed below) is generally designated by the numeral 102. The system 102 may be a client system, for example, since it may obtain audio and / or video streams from a server system (e.g., 120) for the representation of an audio scene and / or visual environment to a user. The client system 102 may also receive metadata from the server system 120, for example, providing side and / or auxiliary information about the audio and / or video streams.

[0079] The system 102 may be associated with (or in some examples may include) a media consumption device (MCD) that actually plays audio and / or video signals to the user. In some examples, the user may wear the MCD.

[0080] The system 102 can execute a request to the server system 120, the request being associated with at least one user's current viewport and / or head orientation (e.g., angular orientation) and / or movement data and / or interaction metadata and / or virtual position data 110. (Several metrics may be provided.) The viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data 110 may be provided in feedback from the MCD to the client system 102, and based on this feedback, the client system 102 may provide the request to the server system 120.

[0081] In some cases, the request (denoted by reference numeral 112) may include the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data 110 (or a displayed or processed version thereof). Based on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data 110, the server system 120 provides the necessary audio and / or video streams and / or metadata. In this case, the server system 120 may have knowledge of the user's position (e.g., in the virtual environment) and can associate the correct streams with the user's position.

[0082] In other cases, the request 112 from the client system 102 may include an explicit request for a particular audio and / or video stream. In this case, the request 112 may be based on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data 110. The client system 102 has knowledge of the audio and video signals that need to be rendered to the user even if the client system 102 does not have the required streams stored within it. The client system 102 may target a particular stream in the server system 120, in an example.

[0083] The client system 102 may be a system for a virtual reality VR, an augmented reality AR, a mixed reality MR, or a 360-degree video environment configured to receive video and audio streams to be played on a media consumption device; The system 102 includes: at least one media video decoder configured to decode a video signal from the video stream to represent a VR, AR, MR, or 360 degree video environment scene to a user; at least one audio decoder 104 configured to decode an audio signal (108) from at least one audio stream 106; The system 102 is configured to request 112 from the server 120 at least one audio stream 106 and / or one audio element of the audio stream and / or one adaptation set based on at least the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data 110.

[0084] It should be noted that in a VR, AR, or MR environment, the user 140 may be meant to be in a particular environment (e.g., a particular room). The environment is described, for example, by a video signal encoded on the server side (not necessarily including the server system 120, but including another encoder that previously encoded a video stream that was then stored in the storage of the server 120). At each moment, in some examples, the user can only enjoy a part of the video signal (e.g., a viewport).

[0085] In general, each environment may be associated with a particular audio scene, which can be understood as the collection of all sounds that are reproduced to the user in a particular environment over a particular period of time.

[0086] Traditionally, environments have been understood as discrete numbers. Therefore, the number of environments has been understood to be finite. For the same reason, the number of audio scenes has been understood to be finite. Therefore, in the prior art, VR, AR, and MR systems are designed as follows: -Users are intended to be in one environment at any given time. So for each environment: o The client system 102 only requests video streams associated with a single environment from the server system 120.

[0087] o The client system 102 only requests audio streams associated with a single scene from the server system 120.

[0088] This approach has become inconvenient.

[0089] For example, all audio streams are delivered to the client system 102 together for each scene / environment, and when the user moves to a different environment, a completely new audio stream needs to be delivered (e.g., when the user goes through a door, meaning a transfer of environment / scene).

[0090] Furthermore, in some cases it can result in unnatural experiences: for example, if a user is close to a wall (such as a virtual wall in a virtual room), they should be able to hear sounds coming from the other side of the wall. However, this experience is not possible in traditional environments: the collection of audio streams associated with the current scene obviously does not include the streams associated with adjacent environments / scenes.

[0091] On the other hand, increasing the bit rate of the audio streams usually improves the user experience. This can lead to further problems. The higher the bit rate, the higher the payload that the server system needs to deliver to the client system 102. For example, if an audio scene contains multiple audio sources (conveyed as audio elements), some of which are close to the user's position and others far from the user's position, the sound sources that are farther away will be hard to hear. Thus, delivering all audio elements at the same bit rate or quality level can result in a very high bit rate. This means inefficient audio stream delivery. If the server system 120 delivers the audio streams at the highest possible bit rate, inefficient delivery occurs, since relevant sounds produced near the user, despite having a low audibility level or low relevance to the overall audio scene, will require a high bit rate as well. Thus, if all audio streams of one scene are delivered at the highest bit rate, the communication between the server system 120 and the client system 102 will unnecessarily increase the payload. If all audio streams of one scene are delivered at a lower bit rate, the user's experience will not be satisfactory.

[0092] Communication issues exacerbate the inconveniences described above: when a user walks through a door, it causes an instantaneous change of environment / scene, requiring the server system 120 to instantly provide all the streams to the client system 102.

[0093] Therefore, the above problem could not be solved in the past.

[0094] However, thanks to the present invention it is possible to solve these problems: the client system 102 makes requests to the server system 120, which may be based on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (and not only based on the environment / scene). Thus, the server system 120 can provide an audio stream that is rendered for each moment, for example, per the user's position.

[0095] For example, if the user is not close to a wall, there is no need for the client system 102 to request streams of the adjacent environment (e.g., the client system 102 may request them only when the user is close to a wall). Furthermore, streams coming from outside the wall may be heard at a lower volume and therefore may have a lower bitrate. In particular, more relevant streams (e.g., streams from audio objects in the current environment) are delivered from the server system 120 to the client system 102 at the highest bitrate and / or highest quality level (so that less relevant streams have lower bitrates and / or quality levels, leaving free bandwidth for more relevant streams).

[0096] Lower quality levels can be obtained, for example, by lowering the bitrate or by processing the audio elements so that less data needs to be transmitted, while the used bitrate per audio signal remains constant. For example, if there are 10 audio objects all located at different positions far away from the user, these objects can be mixed into a fewer number of signals based on the user's position.

[0097] -At locations very far from the user's position (e.g., above a first threshold), the object is mixed into two signals (other numbers are possible based on spatial location and semantics) and delivered as two "virtual objects".

[0098] - At locations close to the user's location (e.g. lower than a first threshold but higher than a second threshold that is smaller than the first), the objects are mixed into 5 signals (based on their spatial location and semantics) and delivered as 5 (other numbers are possible) "virtual objects".

[0099] - At locations very close to the user's location (below the first and second thresholds), the 10 objects are delivered as the 10 audio signals providing the best quality.

[0100] While the highest quality audio signals may all be considered highly significant and audible, the user may be able to identify each object individually. At lower quality levels at greater distances, some audio objects may be less relevant or inaudible, and thus the user will not be able to individually localize the audio signals in space, and therefore lowering the quality level for delivering these audio signals will not result in a decrease in the quality of the user's experience.

[0101] Another example is when the user crosses a door. At a transition location (e.g., a boundary between two different environments / scenes), the server system 120 provides both streams of both scenes / environments, but at a lower bitrate. This is because the user experiences sounds from two different environments (sounds may be merged from different audio streams that were originally associated with different scenes / environments) and the highest quality level of each sound source (or audio element) is not required.

[0102] In view of the above, the present invention makes it possible to go beyond the conventional approach of a discrete number of visual environments and audio scenes, but allows for a progressive representation of different environments / scenes, giving the user a more realistic experience.

[0103] In the following, each visual environment (e.g., virtual environment) is considered to be associated with an audio scene (attributes of an environment can also be attributes of a scene). Each environment / scene can be associated, for example, with a geometric coordinate system (which may be a virtual geometric coordinate system). An environment / scene may have boundaries, so that if the user's position (e.g., virtual position) crosses a boundary, another environment / scene is reached. The boundaries may be based on the coordinate system used. An environment may contain audio objects (audio elements, sound sources) that may be located at some specific coordinates of the environment / scene. For example, with respect to the relative position and / or orientation of the user with respect to the audio objects (audio elements, sound sources), the client system 102 can request different streams and / or the server system 120 can provide different streams (e.g., at higher / lower bitrates and / or quality levels depending on the distance and / or direction).

[0104] More generally, client system 102 may request and / or obtain from server system 120 different streams (e.g., different renditions of the same sound at different bit rates and / or quality levels) based on audibility and / or relevance. Audibility and / or relevance may be determined, for example, based on at least the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data.

[0105] In some examples, there is the possibility to merge different streams. In some cases, there is the possibility to composite, mix, multiplex, overlay, or combine at least two scenes. For example, there is the possibility to use a mixer and / or renderer (e.g., used downstream of multiple decoders, each decoding at least one audio stream), or to perform a stream multiplexing operation, e.g., upstream of the decoding of the streams. In other cases, there may be the possibility to decode various streams and render them with various speaker configurations.

[0106] It should be noted that the present invention does not necessarily reject the notion of visual environments and audio scenes. In particular, in the present invention, audio and video streams associated with a particular scene / environment may be delivered from the server system 120 to the client system 102 when the user enters the environment / scene. Nevertheless, within the same environment / scene, different audio streams and / or audio objects and / or adaptation sets may be requested, addressed and / or delivered. In particular, there are the following possibilities:

[0107] At least a portion of the video data associated with the visual environment is delivered from the server 120 to the client 102 upon the user's entrance into the scene; and / or at least some audio data (streams, objects, adaptation sets, etc.) is delivered to the client system 102 only based on the current (or future) viewport and / or head orientation and / or movement data and / or metadata and / or virtual position and / or user selections / interactions; and / or - (In some cases), some audio data is delivered to the client system 102 based on the current scene (regardless of the current or future position, viewport and / or head orientation and / or movement data and / or metadata and / or virtual position and / or user selection), while the remaining audio data is delivered based on the current or future viewport and / or head orientation and / or movement data and / or metadata and / or virtual position and / or user selection.

[0108] It should be noted that the various elements (server system, client system, MCD, etc.) may represent different hardware devices or elements of the same thing (e.g., the client and MCD may be implemented as part of the same mobile phone, or similarly the client may be located on a PC connected to a secondary screen that constitutes the MCD).

[0109] Working Example One embodiment of the system 102 (client) shown in Fig. 1.1 is configured to receive (audio) streams 106 based on a defined position in an environment (e.g., a virtual environment) that can be understood as being associated with a video and audio scene (hereafter referred to as scene 150). Different positions in the same scene 150 generally imply different streams 106 or different metadata associated with the streams 106 provided (e.g., from a media server 120) to an audio decoder 104 of the system 102. The system 102 is connected to a media consumer device (MCD) from which it receives feedback associated with the user's position and / or virtual position in the same environment. In the following, the user's position in the environment may be associated with a particular viewport enjoyed by the user (e.g., a viewport is intended to be a surface, assumed as a rectangular surface projected onto a sphere, that is displayed to the user).

[0110] In an exemplary scenario, as a user moves within a VR, AR, and / or MR scene 150, audio content can be imagined to be virtually generated by one or more audio sources 152 that may change. The audio sources 152 can be understood as virtual audio sources in the sense that they may point to a location within the virtual environment. The rendering of each audio source is adapted to the user's location (e.g., in a simplified example, the level of the audio source is higher the closer the user is to the audio source's location and lower the further the user is from the audio source). Nevertheless, each audio element (audio source) is encoded into an audio stream that is provided to a decoder. The audio streams can be associated with different locations and / or areas within a scene. For example, an audio source 152 that is not audible in one scene may become audible in the next scene, for example, when a door in the VR, AR, and / or MR scene 150 is opened. The user can then choose to enter a new scene / environment 150 (e.g., a room) and the entire audio scene changes. For the purposes of describing this scenario, we can use the term discrete viewpoints in a space as the discrete locations in a space (or VR environment) at which different audio content is available.

[0111] Generally speaking, the media server 120 can provide a stream 106 associated with a particular scene 150 based on a user's position within the scene 150. The stream 106 can be encoded by at least one encoder 154 and provided to the media server 120. The media server 120 can transmit the stream 113 using communication 113 (e.g., over a communication network). The provision of the stream 113 can be based on a request 112 set by the system 102 based on the user's position 110 (e.g., in the virtual environment). The user's position 110 can also be understood to be associated with a viewport (for each position, there is one single rectangle represented) and a viewpoint (the viewpoint is the center of the viewport) that the user enjoys. Thus, the provision of a viewport may be the same as the provision of a position in some examples.

[0112] The system 102 shown in Fig. 1.2 is configured to receive (audio) streams 113 based on another configuration on the client side. In this exemplary implementation, on the encoding side, multiple media encoders 154 are provided, which can be used to create one or more streams 106 for each available scene 150 associated with one sound scene part of one viewpoint.

[0113] The media server 120 may store multiple audio and video adaptation sets (not shown) that include different encodings of the same audio and video streams at different bit rates. Additionally, the media server may include descriptive information of all adaptation sets that have been created, including availability of all adaptation sets. The adaptation sets may also include information describing the association of one adaptation set to one particular audio scene and / or viewpoint. In this way, each adaptation set can be associated with one of the available audio scenes.

[0114] The adaptation set may further include information describing the boundaries of each audio scene and / or viewpoint, which may include, for example, a complete audio scene or simply individual audio objects. The boundaries of an audio scene may, for example, be defined as the geometric coordinates (e.g., center and radius) of a sphere.

[0115] The client-side system 102 can receive information about the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or any information characterizing the user's virtual position or changes caused by the user's actions. Additionally, the system 102 can also receive information about the availability of all adaptation sets, as well as information describing the association of an adaptation set to an audio scene and / or viewpoint, and / or information describing the "boundaries" of each audio scene and / or viewpoint (e.g., which may include a complete audio scene or only individual objects). For example, such information can be provided as part of the Media Presentation Description (MPD) XML syntax in the case of a DASH delivery environment.

[0116] The system 102 may provide audio signals to a media consumption device (MCD) used for content consumption, which is also responsible for collecting information regarding the user's location and / or orientation and / or direction of movement (or information characterizing changes triggered by the user's actions) as location and transition data 110.

[0117] The viewport processor 1232 may be configured to receive position and transition data 110 from the media consumption device side. The viewport processor 1232 may also receive information about the ROI signaled in the metadata and all information available at the receiving end (system 102). The viewport processor 1232 may then determine which audio viewpoint should be played at a particular moment based on all information received and / or derived from the received and / or available metadata. For example, the viewport processor 1232 may decide to play one complete audio scene and one new audio scene 108 has to be created from all available audio scenes, e.g., only some audio elements of multiple audio scenes are played, while other remaining audio elements of these audio scenes are not played. The viewport processor 1232 may also determine whether a transition between two or more audio scenes needs to be played.

[0118] A selection section 1230 may be provided for selecting, based on information received from the viewport processor 1232, one or more adaptation sets from the available adaptation sets signaled in the information received by the receiving end, where the selected adaptation set completely describes the audio scene to be played at the user's current location. This audio scene may be one complete audio scene defined on the encoding side, or it may be necessary to create a new audio scene from all available audio scenes.

[0119] Further, based on instructions from the viewport processor 1232, when a transition between two or more audio scenes is about to occur, the selection portion can be configured to select one or more adaptation sets from the available adaptation sets signaled in the information received by the receiving end, where the selected adaptation set completely describes the audio scenes that need to be reproduced in the near future (e.g., if a user walks at a certain speed in the direction of the next audio scene, the next audio scene is predicted to be required and selected in advance of playback).

[0120] Furthermore, some adaptation sets corresponding to neighboring locations may be initially selected at a lower bitrate and / or a lower quality level, e.g., a representation encoded at a lower bitrate is selected from the representations available in one adaptation set, and then, based on the change in location, the quality is improved by selecting a higher bitrate for those particular adaptation sets, e.g., a representation encoded at a higher bitrate is selected from the representations available in one adaptation set.

[0121] A downloading and switching portion 1234 may be provided for requesting one or more adaptation sets from the available adaptation sets from the media server based on the instruction received from the selection portion, and is configured to receive one or more adaptation sets from the available adaptation sets from the media server and extract metadata information from all received audio streams.

[0122] A metadata processor 1236 may be provided for receiving information from download and switching information for the received audio streams, which may include audio metadata corresponding to each received audio stream. The metadata processor 1236 may also be configured to process and manipulate the audio metadata associated with each audio stream 113 based on information received from the viewport processor 1232, which may include information regarding the user's position and / or orientation and / or movement direction 110, in order to select / activate the necessary audio elements 152 constituting a new audio scene, as indicated by the viewport processor 1232, so as to be able to merge all audio streams 113 into a single audio stream 106.

[0123] The stream muxer / merger 1238 may be configured to merge all selected audio streams into one audio stream 106 based on information received from the metadata processor 1236, which may include modified and processed audio metadata corresponding to all received audio streams 113.

[0124] The media decoder 104 is configured to receive and decode at least one audio stream for playback of a new audio scene, as indicated by the viewport processor 1232, based on information regarding the user's position and / or orientation and / or direction of movement.

[0125] In another embodiment, the system 102 shown in FIG. 1.7 may be configured to receive audio streams 106 at different audio bitrates and / or quality levels. The hardware configuration of this embodiment is the same as that of FIG. 1.2. At least one visual environment scene 152 can be associated with at least one plurality of N audio elements (N>=2), and each audio element is associated with a position and / or region within the visual environment. At least one plurality of N audio elements 152 are provided in at least one representation at a high bitrate and / or quality level, at least one plurality of N audio elements 152 are provided in at least one representation at a low bitrate and / or quality level, and at least one representation is obtained by processing the N audio elements 152 to obtain a smaller number M (M<N) of audio elements 152 associated with a position or region close to the position or region of the N audio elements 152.

[0126] The processing of the N audio elements 152 may be, for example, a simple addition of the audio signals, or an active downmix based on their spatial positions 110, or the rendering of the audio signals to a new virtual position located between the audio signals using their spatial positions. The system may be configured to request a representation at a higher bitrate and / or quality level for an audio element if the audio element is more relevant and / or more audible at the current virtual position of the user in the scene, and the system may be configured to request a representation at a lower bitrate and / or quality level for an audio element if the audio element is less relevant and / or less audible at the current virtual position of the user in the scene.

[0127] FIG. 1.8 illustrates an example of a system (which may be system 102) for a virtual reality VR, augmented reality AR, mixed reality MR, or 360-degree video environment configured to receive a video stream 1800 and an audio stream 106 to be played on a media consumption device; The system 102 includes: at least one media video decoder 1804 configured to decode a video signal 1808 from the video stream 1800 to represent a VR, AR, MR, or 360 degree video environment to a user; and at least one audio decoder 104 configured to decode an audio signal 108 from at least one audio stream 106.

[0128] The system 102 may be configured to request (112) at least one audio stream 106 and / or one audio element of the audio stream and / or one adaptation set from a server (e.g., 120) based on at least a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data 110 (e.g., provided as feedback from the media consumption device 180).

[0129] The system 102 may be the same as the system 102 of Figs. 1.1-1.7 and / or may capture the scenarios of Figs. 2a onwards.

[0130] This example also refers to a method for a virtual reality (VR), augmented reality (AR), mixed reality (MR), or 360-degree video environment configured to receive video and audio streams to be played on a media consumption device (e.g., a playback device), the method comprising: decoding a video signal from the video stream for presentation to a user of a VR, AR, MR, or 360 degree video environment scene; decoding an audio signal from the audio stream; and requesting and / or obtaining at least one audio stream from the server based on the user's current viewport and / or position data and / or head orientation and / or movement data and / or metadata and / or virtual position data and / or metadata.

[0131] Case 1 Different scenes / environments 150 generally imply the receipt of different streams 106 from the server 120. However, the streams 106 received by the audio decoder 104 may also be conditioned by the position of the user in the same scene 150.

[0132] At a first (start) time point (t=t1) shown in FIG. 2a, the user is, for example, located in the scene 150 and has a first defined position in the VR environment (or AR environment, or MR environment). In a Cartesian XYZ coordinate system (e.g., horizontal), the user's first viewport (position) 110′ is located at coordinates x′ u and y' u (axis Z is here oriented out of the paper). In this first scene 150, two audio elements 152-1 and 152-2 are positioned with coordinates x'1 and y'1 for audio element 1 (152-1) and x'2 and y'2 for audio element 2 (152-2), respectively. The user's distance d'1 to audio element 1 (152-1) is less than the user's distance d'2 to audio element 2 (152-1). All user position (viewport) data is transmitted from the MCD to the system 102.

[0133] At a second exemplary time point (t=t2) shown in FIG. 2b, the user is positioned, for example, within the same scene 150, but at a second, different location. In a Cartesian XY coordinate system, the user's second viewport (location) 110″ is at a new coordinate x″ u and y”u (axis Z is now oriented out of the paper), where the user's distance d''1 from audio element 1 (152-1) is greater than the user's distance d''2 from audio element 2 (152-2). All user position (viewport) data is sent back to the system 102 from the MCD.

[0134] A user equipped with the MCD to visualize a particular viewport within a 360-degree environment may, for example, be listening through headphones, and may enjoy different sound playback for the different positions shown in Figures 2a and 2b of the same scene 150.

[0135] Any position and / or transition and / or viewport and / or virtual position and / or head orientation and / or movement data within the scene, e.g., of FIG. 2a-b, can be periodically (e.g., in feedback) transmitted from the MCD to the system 102 (client) as a signal 110. The client can retransmit the position and transition data 110′ or 110″ (e.g., viewport data) to the server 120. The client 102 or server 120 can determine the audio streams 106 required to play the correct audio scene at the current user position based on the position and transition data 110′ or 110″ (e.g., viewport data). The client can determine and transmit a request 112 for the corresponding audio streams 106, and the server 120 can be configured to deliver the streams 106 accordingly depending on the position information provided by the client (system 102). Alternatively, the server 120 may determine and deliver the streams 106 accordingly depending on the position information provided by the client (system 102).

[0136] The client (system 102) may request the transmission of the streams to be decoded to represent the scene 150. In some examples, the system 102 may transmit information regarding the highest quality level to be reproduced in the MCD (in other examples, it is the server 120 that decides the quality level to be reproduced in the MCD based on the user's position in the scene). In response, the server 120 may select one of a number of representations associated with the represented audio scene and deliver at least one stream 106 according to the user's position 110' or 110". Thus, the client (system 102) may be configured to deliver the audio signal 108 to the user, for example, via the audio decoder 104, and reproduce sounds related to the user's actual (effective) position 110' or 110" (an adaptation set 113 may be used; for example, different variants of the same stream at different bit rates may be used for different positions of the user).

[0137] Streams 106 (which may be pre-processed or generated on the fly) can be sent to a client (system 102) and can be configured for multiple viewpoints associated with a particular sound scene.

[0138] It should be noted that different qualities (e.g., different bit rates) may be provided for different streams 106 according to a particular position (e.g., 110' or 110") of the user in the (e.g., virtual) environment. For example, in the case of multiple audio sources 152-1 and 152-2, each audio source 152-1 and 152-2 may be associated with a particular position in the scene 150. The closer the user's position 110' or 110' is to the first audio source 152-1, the more quality (e.g., different bit rates) may be provided for the different streams 106. The further the user's position 110 is from the second audio source 152-2, the lower the required resolution of the stream 106 associated with the second audio source 152-2. This exemplary case can be applied to audio element 2 (152-2) of FIG. 2a as well as audio element 1 (152-1) of FIG. 2b. The further the user's position 110 is from the second audio source 152-2, the lower the required resolution of the stream 106 associated with the second audio source 152-2. This exemplary case can be applied to audio element 2 (152-2) of FIG. 2a as well as audio element 1 (152-1) of FIG. 2b.

[0139] In effect, firstly, closer audio sources are heard at a higher level (and therefore provided at a higher bit rate), and secondly, more distant audio sources are heard at a lower level (which can make it possible to request a lower resolution).

[0140] Thus, based on the location 110' or 110" in the environment served by the client 102, the server 120 can provide different streams 106 at different bitrates (or other qualities). Based on the fact that audio elements that are farther away do not require a high quality level, the overall user quality experience is maintained even when delivered at a lower bitrate or quality level.

[0141] Thus, different quality levels can be used for some audio elements at different user locations while maintaining the quality of the experience.

[0142] Without this solution, all streams 106 would be provided from the server 120 to the client at the highest bit rate, which would increase the payload of the communication channel from the server 120 to the client.

[0143] Case 2 FIG. 3 (Case 2) shows one embodiment of another exemplary scenario (represented in a vertical plane XZ of space XYZ, with axis Y represented as entering the paper), where a user moves in a first VR, AR, and / or MR scene A (150A), opens a door, and walks through it (transition 150AB), which implies a transition of audio from the first scene 150A at time t1, through a temporal position (150AB) at time t2, to the next (second) scene B (150B) at time t3.

[0144] At time t1, the user may be at position x1 in the x-direction of a first VR, AR and / or MR scene. At time t3, the user may be in a different second VR, AR and / or MR scene B (150B) at position x3. At moment t2, the user may be at transition position 150AB while opening and walking through a door (e.g., a virtual door). The transition thus refers to the transition of audio information from the first scene 150A to the second scene 150B.

[0145] In this situation, the user is changing his / her position 110, for example, from a first VR environment (characterized by a first viewpoint (A) as shown in FIG. 1.1) to a second VR environment (characterized by a second viewpoint (B) as shown in FIG. 1.1). In a particular case, for example, during the transition through a door at position x2 in the x-direction, some audio elements 152A and 152B may be present in both viewpoints (positions A and B).

[0146] The user (equipped with an MCD) has changed position 110 (x1-x3) towards the door, which means that at transition position x2, the audio element belongs to both the first scene 150A and the second scene 150B. The MCD sends the new position and transition data 110 to the client, which resends it to the media server 120. The user may now be able to hear the appropriate audio source defined by the intermediate position x2 between the first position x1 and the second position x3.

[0147] Any position and transition from a first position (x1) to a second position (x3) is periodically (e.g. continuously) transmitted from the MCD to the client. The client 102 can retransmit the position and transition data 110 (x1-x3) to the media server 120, which is configured to deliver one dedicated item, such as a new set of pre-processed streams 106 in the form of an actual adaptation set 113', depending on the received position and transition data 110 (x1-x3).

[0148] The media server 120 can select one of a number of representations associated with the aforementioned information regarding the MCD's ability to display the highest bitrate, as well as the user's location and transition data 110 (x1-x3) during the transition from one location to another. (In this situation, an adaptation set can be used. The media server 120 can determine which adaptation set 113' best represents the user's virtual transition without interfering with the MCD's rendering capabilities.) Thus, the media server 120 can deliver the dedicated stream 106 according to the location transition (e.g., as a new adaptation set 113'). The client 102 may be configured to deliver the audio signal 108 to the user 140 accordingly, e.g., via the media audio decoder 104.

[0149] The stream 106 (generated on the fly and / or pre-processed) may be transmitted to the client 102 in adaptation sets 113' implemented periodically (eg, continuously).

[0150] When the user walks through the door, the server 120 can send both the stream 106 of the first scene 150A and the stream 106 of the second scene 150B in order to mix or multiplex or compose or play these streams 106 simultaneously to give the user a realistic impression. Thus, based on the user's position 110 (e.g., "position corresponding to the door"), the server 120 sends different streams 106 to the client.

[0151] Even in this case, since the different streams 106 are heard at the same time, they may have different resolutions and be transmitted from the server 120 to the client at different resolutions. When the user completes the transition and is in the second (location) scene 150A (and has closed the door behind him), the server 120 may reduce or refrain from transmitting the streams 106 of the first scene 150 (if the server 120 is already providing the streams to the client 102, the client 102 may decide not to use them).

[0152] Case 3 FIG. 4 (Case 3) illustrates an embodiment with another exemplary scenario (represented in a vertical plane XZ of space XYZ, with axis Y represented as entering the paper), meaning audio transitions from one first position at time t1 to a second position in the first scene 150A at time t2 as the user moves through the VR, AR and / or MR scene 150A. The user at the first position may be far from the wall at time t1, a distance d1 from the wall, and close to the wall at time t2, a distance d2 from the wall, where d1>d2. At distance d1, the user hears only source 152A in scene 150A, but can also hear source 152B in scene 150B beyond the wall.

[0153] When the user is at the second location (d2), the client 102 transmits data regarding the user's location 110 (d2) to the server 120 and receives from the server 120 not only the audio stream 106 of the first scene 150A but also the audio stream 106 of the second scene 150B. For example, based on metadata provided by the server 120, the client 102 may cause the stream 106 of the second scene 150B to be played at a lower volume (through the wall), for example, via the decoder 104.

[0154] In this case, the stream 106 of the second scene 150B may still have a lower bitrate (quality) and therefore need to reduce the transmission payload from the server 120 to the client. In particular, the position 110 (d1, d2) of the client (and / or viewport) defines the audio stream 106 provided by the server 120.

[0155] For example, the system 102 may be configured to acquire a stream associated with a first current scene (150A) associated with a first current environment, and the system 102 may acquire a stream associated with a first current scene (150B) associated with a first current environment, and the system 102 may acquire a stream associated with a first current scene (150C) associated with a first current environment, and the system 102 may acquire a stream associated with a first current scene (150D) associated with a first current environment, and the system 102 may acquire a stream associated with a first current scene (150E) associated with a first current environment, and the system 102 may acquire a stream associated with a first current scene (150F) associated with a first current environment, and the system 102 may acquire a stream associated with a first current scene (150G ... <d しきい値 ), the system 102 further obtains an audio stream associated with a second, adjacent and / or proximate environment associated with the second scene (150B).

[0156] Case 4 5a and 5b show an embodiment with another exemplary scenario (represented in a horizontal plane XY in space XYZ, with axis Z represented as coming out of the paper), where a user is located in the same VR, AR, and / or MR scene 150, but at different moments in time, with different distances to, for example, two audio elements.

[0157] At a first time instant t=t1 shown in FIG. 5a, the user is, for example, located at a first position. In this first position, a first audio element 1 (152-1) and a second audio element 2 (152-2) are located (for example, substantially) at distances d1 and d2, respectively, from the user with the MCD. In this case, both distances d1 and d2 are within a defined threshold distance d しきい値 , and thus the system 102 is configured to group both audio elements into a single virtual source 152-3. The position and properties (such as spatial extent) of the single virtual source can be calculated, for example, based on the positions of the original two sources in such a way as to mimic as closely as possible the original sound field generated by the two sources (for example, two well-localized point sources can be reproduced as a single source in the middle of the distance between them). The user position data 110 (d1, d2) can be transmitted from the MCD to the system 102 (client) and subsequently to the server 120, which can decide to transmit the appropriate audio stream 106 to be rendered by the server system 120 (in other embodiments, it is the client 102 that decides which stream to transmit from the server 120). By grouping both audio elements into a single virtual source 152-3, the server 120 can select one of a number of representations associated with the aforementioned information. (For example, it is possible to deliver a dedicated stream 106 accordingly, and an adaptation set 113' associated with, for example, one single channel.) Thus, via MCD, a user can receive an audio signal transmitted from a single virtual audio element 152-3 positioned between real audio elements 1 (152-1) and 2 (152-2).

[0158] At a second instant t=t2, as shown in FIG. 5b, the user is, for example, located within the same scene 150 and has a second defined position in the same VR environment as FIG. 5a. In this second position, the two audio elements 152-1 and 152-2 are located (for example, virtually) at distances d3 and d4 from the user, respectively. Both distances d3 and d4 are within a threshold distance d しきい値 The time may be shorter, and therefore the grouping of the audio elements 152-1 and 152-2 into a single virtual source 152-3 is no longer used. The user position data is transmitted from the MCD to the system 102 and subsequently to the server 120, which can decide to send another suitable audio stream 106 to be rendered by the system server 120 (in other embodiments, this decision is made by the client 102). By avoiding grouping the audio elements, the server 120 can select a different representation associated with the aforementioned information and accordingly deliver a dedicated stream 106 with an adaptation set 113' associated to a different channel for each audio element. As a result, the user can receive, via the MCD, an audio signal 108 transmitted from two different audio elements 1 (152-1) and 2 (152-2). Thus, the closer the user's position 110 is to the audio sources 1 (152-1) and 2 (152-2), the higher the required quality level of the stream associated with the audio source should be selected.

[0159] Indeed, as shown in Fig. 5b, the closer audio sources 1 (152-1) and 2 (152-2) are to the user, the higher the level needs to be adjusted and therefore the audio signal 108 is rendered at a higher quality level. In contrast, the more remotely located audio sources 1 and 2 depicted in Fig. 5b are rendered at a lower quality level, for example, as they need to be heard at a lower level when reproduced by a single virtual source.

[0160] In a similar configuration, multiple audio elements are placed in front of the user, all of them at a distance greater than a threshold distance from the user. In one embodiment, two groups of five audio elements may be combined into two virtual sources each. The user position data is transmitted from the MCD to the system 102 and subsequently to the server 120, which can decide to transmit the appropriate audio stream 106 to be rendered by the system server 120. By grouping all ten audio elements into only two single virtual sources, the server 120 can select one of the multiple representations associated with the aforementioned information and accordingly deliver a dedicated stream 106 with, for example, an adaptation set 113' associated with the two single audio elements. As a result, the user can receive, via the MCD, audio signals transmitted from two different virtual audio elements placed in the same placement area as the real audio elements.

[0161] At a later moment, the user approaches a large number (10) of audio elements. In this subsequent scene, all audio elements are within the threshold distance d しきい値Since the user is located at a smaller distance, the system 102 is configured to terminate the grouping of the audio elements. New user position data is transmitted from the MCD to the system 102 and subsequently to the server 120, which can decide to transmit another suitable audio stream 106 to be rendered by the server system 120. By not grouping the audio elements, the server 120 can select a different representation associated with the aforementioned information and accordingly deliver a dedicated stream 106 with an adaptation set 113' associated with a different channel for each audio element. As a result, the user can receive, via the MCD, audio signals transmitted from ten different audio elements. Thus, the closer the user's position 110 is to the audio source, the higher the required resolution of the stream associated with the audio source must be selected.

[0162] Case 5 FIG. 6 (Case 5) illustrates a user 140 at one position in a single scene 150 wearing a media consumer device (MCD) that can be oriented in three exemplary different directions (each associated with a different viewport 160-1, 160-2, 160-3). These directions illustrated in FIG. 6 may have directions (e.g., angular orientations) in a polar and / or Cartesian XY coordinate system that point to a first viewpoint 801 at, for example, 180° at the bottom of FIG. 6, a second viewpoint 802 located at, for example, 90° at the right side of FIG. 6, and a third viewpoint 803 located at, for example, 0° at the top of FIG. 6. Each of these viewpoints is associated with an orientation of the user 140 wearing the media consumer device (MCD), such that a user located in the center is provided with a particular viewport displayed by the MCD that renders a corresponding audio signal 108 according to the orientation of the MCD.

[0163] In this particular VR environment, a first audio element s1 (152) is located in a first viewport 160-1, e.g., near a viewpoint located at 180°, and a second audio element s2 (152) is located in a third viewport 160-3, e.g., near a viewpoint located at 180°. Prior to changing his / her orientation, the user 140 experiences that in a first orientation towards the viewpoint 801 (viewport 160-1), the sound associated with the user's actual (effective) position is louder from audio element s1 than from audio element s2.

[0164] By changing the user's orientation, the user 140 experiences that in a second orientation, towards the viewpoint 802, the sound associated with the user's actual position 110 comes from the side at approximately the same volume from both audio elements s1 and s2.

[0165] Finally, by changing the user's orientation, user 140 can experience the sound associated with audio element 2 as being louder than the sound associated with audio element s1 in a third orientation toward viewpoint 801 (viewport 160-3) (in reality, the sound from audio element 2 arrives from the front and the sound from audio element 1 arrives from the rear).

[0166] Thus, different viewports and / or orientations and / or virtual position data may be associated with different bitrates and / or qualities.

[0167] Further Cases and Examples FIG. 7A illustrates an embodiment of a method for receiving an audio stream by a system in the form of a series of operational steps in the figure. At any moment, a user of the system 102 is associated with the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position. At a particular moment, the system can determine, in step 701 of FIG. 7A, the audio elements to be played based on the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position. Thus, in the next step 703, the relevance and audibility level of each audio element can be determined. As described above in FIG. 6, a VR environment can have different audio elements located in a particular scene 150, close to the user or further away, but also with a particular orientation in the 360 ​​degrees around. All these factors determine the relevance and audibility level of each audio element.

[0168] In a next step 705 , the system 102 may request audio streams according to the determined relevance and audibility levels for each of the audio elements from the media server 120 .

[0169] In a next step 707, the system 102 can receive audio streams 113 appropriately prepared by the media server 120, with streams of different bit rates reflecting the relevance and audibility levels determined in the previous step.

[0170] In a next step 709, the system 102 (e.g., an audio decoder) can decode the received audio stream 113, whereby in step 711 a particular scene 150 is reproduced (e.g., by MCD) according to the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position.

[0171] 7B illustrates an interaction between the media server 120 and the system 102 according to the sequence of operations described above. At a particular moment, the media server can transmit an audio stream 750 at a lower bit rate according to the above-determined lower relevance and audibility level of the relevant audio elements of the aforementioned scene 150. At a subsequent moment 752, the system can determine that an interaction or a change in location data has occurred. Such an interaction can result, for example, from a change in location data in the same scene 150, or, for example, from a user activating a door handle while attempting to enter a second scene separated from the first scene by a door provided by the door handle.

[0172] Changes in the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position can result in a request 754 being sent by the system 102 to the media server 120. The request can reflect a higher relevance and audibility level of the relevant audio elements determined for the subsequent scene 150. In response to the request 754, the media server transmits a stream 756 at a higher bitrate to enable a plausible, realistic playback of the scene 150 at the current user's virtual position by the system 102.

[0173] 8A also shows another embodiment of a method for receiving an audio stream by a system in the form of a sequence of operational steps in the figure. At a certain instant 801, a determination of a first current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position may be performed. By deducting the positive cases, a request for a stream associated with a first position defined by a low bit rate may be prepared and transmitted by the system 102 in step 803.

[0174] A decision step 805 with three different outcomes may be performed at subsequent moments. One or two defined thresholds may be associated in this step to determine a predictive decision regarding, for example, a subsequent viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position. Thus, a comparison with a first and / or a second threshold can be performed regarding the probability of a change to a second position, resulting in, for example, three different subsequent steps being performed.

[0175] For example, for a result reflecting a very low probability (eg, associated with a comparison to the first predetermined threshold value above), a new comparison step 801 is performed.

[0176] In an outcome reflecting a low probability (e.g., higher than the first predetermined threshold, but in the example, higher than the first threshold and lower than the second predetermined threshold), a request for a low bit rate audio stream 113 may occur in step 809.

[0177] An outcome reflecting a high probability (e.g., higher than a second predetermined threshold) may result in execution of a request for a high bit rate audio stream 113 in step 807. Thus, the subsequent step executed after executing steps 807 or 809 may again be decision step 801.

[0178] FIG. 8B illustrates an interaction between the media server 120 and the system 102 according to only one of the sequences of the aforementioned operational diagrams. At a certain moment, the media server can transmit an audio stream 850 at a low bit rate according to the aforementioned determined low relevance and audibility level of the audio elements of the aforementioned scene 150. The system can determine that at a subsequent moment 852, an interaction is to occur predictively. A predictive change in the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position can result in an appropriate request 854 being transmitted by the system 102 to the media server 120. This request can reflect one of the above cases of a high probability of reaching a second position associated with a high bit rate according to the audibility level of the audio elements required for the respective subsequent scene 150. In response, the media server transmits a stream 856 at a higher bit rate, enabling a plausible and realistic reproduction of the scene 150 by the system 102 at the current user's virtual position.

[0179] The system 102 shown in Fig. 1.3 is configured to receive the audio stream 113 based on another configuration on the client side, where the system architecture can use a discrete viewpoint based on a solution using multiple audio decoders 1320, 1322. On the client side, the system 102 can for example embody parts of the system described in Fig. 1.2 additionally or alternatively with multiple audio decoders 1320, 1322, which can be configured to decode individual audio streams, as indicated by the metadata processor 1236, for example, with some audio elements deactivated.

[0180] The system 102 may be provided with a mixer / renderer 1238 that is configured to play the final audio scene based on information about the user's position and / or orientation and / or direction of movement, i.e., for example, disabling or not rendering some audio elements that cannot be heard in that particular location.

[0181] The following embodiments shown in Figures 1.4, 1.5 and 1.6 are based on independent adaptation sets for discrete viewpoints with flexible adaptation sets. When a user moves in a VR environment, the audio scene may change continuously. To ensure a good audio experience, all audio elements that compose an audio scene at a given time need to be made available to the media decoder, which can utilize the position information to create the final audio scene.

[0182] If the content is pre-encoded, then at some pre-defined locations, the system can accurately play back the audio scenes at these specific locations, under the assumption that these audio scenes do not overlap and the user can "jump / switch" from one location to the next.

[0183] However, if a user "walks" from one location to the next, audio elements of two (or more) audio scenes can be heard simultaneously. A solution to this use case is provided in the previous example system that does not rely on a mechanism provided to decode multiple audio streams (either using a muxer with a single media decoder or multiple media decoders with additional mixers / renderers), but rather requires the client to provide audio streams that describe a complete audio scene.

[0184] An optimization is provided below by introducing the concept of common audio elements among multiple audio streams.

[0185] Description of Aspects and Examples Solution 1: Independent adaptation set of discrete positions (viewpoints).

[0186] One way to solve the above problem is to use a completely independent set of adaptations for each location. To better understand this solution, we will use Figure 1.1 as an example scenario. In this example, we use three different and separate viewpoints (composed of three different audio scenes) to create a complete VR environment in which the user can move around. Thus, · Several independent or overlapping audio scenes are encoded into several audio streams. For each audio scene, one main stream can be used, or one main stream and additional auxiliary streams depending on the use case (e.g. some audio objects containing different languages ​​can be encoded into independent streams for efficient delivery). In the provided example, audio scene A is encoded into two streams (A1 and A2), audio scene B into three streams (B1, B2 and B3) and audio scene C into three streams (C1, C2 and C3). Note that audio scenes A and B share some common elements (two audio objects in this example). Since all scenes need to be complete and independent (e.g. for independent playback on non-VR playback devices), the common elements need to be encoded twice for each scene.

[0187] All audio streams are encoded at different bitrates (i.e. different representations), allowing efficient bitrate adaptation depending on the network connection (i.e. users with fast connections are served a high-bitrate coded version, while users with slower network connections are delivered a lower-bitrate version).

[0188] The audio streams are stored in a media server, and for each audio stream, different encodings with different bitrates (i.e. different representations) are grouped into one adaptation set, and the appropriate data is signaled for the availability of all created adaptation sets.

[0189] Furthermore, in addition to the adaptation sets, the media server receives information about the location of the "boundary" of each audio scene and its relationship to each adaptation set (e.g. including a complete audio scene or only individual objects). In this way, each adaptation set can be associated with one of the available audio scenes. The boundary of one audio scene may for example be defined as the geometric coordinates of a sphere (e.g. center and radius).

[0190] o Each adaptation set also contains descriptive information about where sound scenes or audio elements are active. For example, if one auxiliary stream contains one or more objects, an adaptation set may contain information such as where the objects can be heard (e.g., the coordinates and radius of the center of a sphere).

[0191] The media server provides information to the client (e.g., a DASH client) about the location of the "boundaries" associated with each adaptation set. For example, in the case of a DASH delivery environment, this may be embedded in the Media Presentation Description (MPD) XML syntax.

[0192] The client receives information about the user's position and / or orientation and / or direction of movement (or information characterizing changes triggered by the user's actions).

[0193] The client receives information about each Adaptation Set, and based on this and the user's position and / or orientation and / or direction of movement (or information characterizing changes triggered by the user's actions, such as x, y, z coordinates or yaw, pitch, roll values), the client selects one or more Adaptation Sets that completely describe the audio scene to be played at the user's current location.

[0194] The client requests one or more adaptation sets. Additionally, the client may select more adaptation sets that fully describe the multiple audio scenes and use the audio streams corresponding to the multiple audio scenes to create a new audio scene that should be played at the user's current location (e.g., the user is walking in the VR environment and at some point is in between (or in a location where the two audio scenes have the effect of being heard).

[0195] o Once the audio streams are available, multiple media decoders may be used to decode the individual audio streams and additional mixers / renderers 1238 may be used to play the final audio scene based on information about the user's position and / or orientation and / or direction of movement (i.e., for example, disabling or not rendering some of the audio elements that cannot be heard at that particular location).

[0196] Alternatively, by using the metadata processor 1236 to manipulate audio metadata associated with all audio streams based on information about the user's position and / or orientation and / or direction of movement, · Select / enable the required Audio Elements 152 that will compose the new Audio Scene.

[0197] Also allows merging all audio streams into a single audio stream.

[0198] The media server delivers the required set of adaptations.

[0199] Alternatively, the client provides information about the user's positioning to the media server, and the media server provides instructions regarding the required adaptation set.

[0200] Figure 1.2 shows another implementation of such a system.

[0201] Encoding side o Multiple media encoders that can be used to create one or more audio streams for each available audio scene associated with one sound scene part of one viewpoint o Multiple media encoders that can be used to create one or more video streams for each available video scene associated with one video scene part of one viewpoint. For simplicity, the video encoders are not shown in the diagram.

[0202] o A media server that stores multiple audio and video adaptation sets that contain different encodings of the same audio and video streams at different bitrates (i.e., different representations). Additionally, the media server stores description information for every adaptation set, which may include:

[0203] Availability of all created adaptation sets.

[0204] Information describing the association of one adaptation set with one audio scene and / or viewpoint. In this way, each adaptation set can be associated with one of the available audio scenes.

[0205] Information describing the "boundaries" of each audio scene and / or viewpoint (which may, for example, contain complete audio scenes or only individual objects). The boundary of one audio scene may, for example, be defined as the geometric coordinates of a sphere (e.g. center and radius).

[0206] On the client side, a system (client system) that includes any of the following:

[0207] o A receiver capable of receiving: Information about the user's position and / or orientation and / or direction of movement (or information characterizing changes triggered by the user's actions) Information about the availability of all Adaptation Sets, as well as information describing the association of an Adaptation Set with an audio scene and / or viewpoint, and / or information describing the "boundaries" of each audio scene and / or viewpoint (which may, for example, include complete audio scenes or only individual objects). For example, such information may be provided as part of the Media Presentation Description (MPD) XML syntax in the case of a DASH delivery environment.

[0208] On the media consumption device side, used for content consumption (e.g., based on an HMD), the media consumption device is also responsible for collecting information about the user's position and / or orientation and / or direction of movement (or information characterizing changes triggered by the user's actions).

[0209] o The viewport processor 1232 can be configured as follows:

[0210] · Receive information about the current viewport, including the user's position and / or orientation and / or direction of movement, from the media consumption device (or information characterizing changes triggered by the user's actions).

[0211] Receive information about the ROI signaled in the metadata (video viewport signaled in the OMAF specification).

[0212] · Receive all information available at the receiving end.

[0213] Deciding which audio / video viewpoint to play at a particular moment based on all information received and / or derived from received and / or available metadata. For example, the viewport processor 1232 may decide:

[0214] A complete audio scene is recreated.

[0215] You need to create one new Audio Scene from all available Audio Scenes (for example, only some audio elements from multiple Audio Scenes are played, while other remaining audio elements from these Audio Scenes are not played).

[0216] You need to recreate a transition between two or more audio scenes.

[0217] o A selection portion 1230 configured to select, based on information received from the viewport processor 1232, one or more adaptation sets from the available adaptation sets signaled in the information received by the receiving end. The selected adaptation set completely describes the audio scene to be played at the user's current location. This audio scene can be one complete audio scene defined on the encoding side or a new audio scene has to be created from all available audio scenes.

[0218] Further, based on instructions from the viewport processor 1232, if a transition between two or more audio scenes is about to occur, the selection portion 1230 can be configured to select one or more adaptation sets from the available adaptation sets signaled in the information received by the receiving end, where the selected adaptation set completely describes the audio scenes that need to be reproduced in the near future (e.g., if a user walks at a certain speed in the direction of the next audio scene, the next audio scene is predicted to be needed and selected prior to playback).

[0219] Furthermore, some adaptation sets corresponding to neighboring locations may be selected initially at a lower bitrate (i.e., a representation encoded at a lower bitrate is selected from the representations available in one adaptation set), and then, based on the change in location, the quality is improved by selecting a higher bitrate for these particular adaptation sets (i.e., a representation encoded at a higher bitrate is selected from the representations available in one adaptation set).

[0220] o The download and switching part, which can be configured as follows: · Requesting one or more of the available adaptation sets from the media server 120 based on the indication received from the selection portion 1230.

[0221] · Receive one or more adaptation sets from the available adaptation sets from the media server 120 (i.e. one representation out of all representations available within each adaptation set).

[0222] Extract metadata information from all received audio streams A metadata processor 1236 that can be configured as follows: Receiving information from download and switching information about the received audio streams, which may include audio metadata corresponding to each received audio stream.

[0223] by processing and manipulating audio metadata associated with each audio stream based on information received from the viewport processor 1232, which may include information regarding the user's position and / or orientation and / or direction of movement; · Select / activate the required audio elements 152 that compose the new audio scene, as indicated by the viewport processor 1232.

[0224] All audio streams can be merged into a single audio stream.

[0225] o The stream muxer / merger 1238 may be configured to merge all selected audio streams into one audio stream based on information received from the metadata processor 1236, which may include modified and processed audio metadata corresponding to all received audio streams.

[0226] o The media decoder is configured to receive and decode at least one audio stream for playback of a new audio scene, as indicated by the viewport processor 1232, based on information regarding the user's position and / or orientation and / or direction of movement.

[0227] FIG. 1.3 shows a system including a system (client system) which may embody, on the client side, for example, part of the system described in FIG. 1.2, which additionally or alternatively includes:

[0228] · Multiple media decoders can be configured to decode individual audio streams as indicated by the Metadata Processor 1236 (e.g. some audio elements are deactivated).

[0229] The mixer / renderer 1238 may be configured to play back the final audio scene based on information about the user's position and / or orientation and / or direction of movement (i.e., for example, disabling or not rendering some audio elements that cannot be heard at that particular location).

[0230] Solution 2 Figures 1.4, 1.5 and 1.6 show an example based on solution 2 of the present invention (which may be an embodiment of the examples of Figures 1.1 and / or 1.2 and / or 1.3): Independent adaptation sets of discrete positions (viewpoints) with flexible adaptation sets.

[0231] As the user moves through the VR environment, the audio scene 150 may change continuously. To ensure a good audio experience, all audio elements 152 that make up the audio scene 150 at a particular point in time need to be available to the media decoder, which can utilize the position information to create the final audio scene.

[0232] If the content is pre-encoded, then at some pre-defined locations, the system can accurately play back the audio scenes at these specific locations, under the assumption that these audio scenes do not overlap and the user can "jump / switch" from one location to the next.

[0233] However, if a user "walks" from one location to the next, audio elements 152 of two (or more) audio scenes 150 may be heard simultaneously. A solution to this use case is provided in the previous system example that does not rely on the mechanism provided to decode multiple audio streams (either using a muxer with a single media decoder or multiple media decoders with additional mixer / renderers 1238) and requires providing audio streams that describe a complete audio scene 150 to the client / system 102.

[0234] By introducing the concept of common audio elements 152 between multiple audio streams, optimizations are provided below.

[0235] Figure 1.4 shows an example where different scenes share at least one audio element (audio object, sound source, etc.). Thus, the client 102 may receive, for example, one main stream 106A associated with only one scene A (e.g., associated with the environment where the user is currently located) and associated with an object 152A, and one auxiliary stream 106B shared by a different scene B (e.g., a stream within the boundary between scene A where the user is currently located and adjacent or neighboring stream B sharing object 152B) and associated with object 152B.

[0236] Therefore, as shown in Figure 1.4, Several independent or overlapping audio scenes are encoded into several audio streams. The audio streams 106 are created in the following way:

[0237] o For each audio scene 150, one main stream can be created that contains only the audio elements 152 that are part of the respective audio scene and that are not part of other audio scenes; and / or o For all audio scenes 150 that share an audio element 152, the common audio element 152 is only encoded in the auxiliary audio stream associated with only one of the audio scenes, and appropriate metadata information is created indicating the association with the other audio scenes; or, in other words, additional metadata indicates the possibility that some audio streams may be used together with multiple audio scenes; and / or o Depending on the use case, additional auxiliary streams may be created (e.g., some audio objects containing different languages ​​may be encoded into independent streams for efficient delivery).

[0238] In the embodiments provided, Audio scene A is encoded as: Main audio stream (A1, 106A), Auxiliary audio streams (A2, 106B) Metadata information capable of indicating that some audio elements 152B of audio scene A are not encoded in these audio streams A but in an auxiliary stream A2 (106B) belonging to a different audio scene (audio scene B). Audio scene B is encoded as: Main audio stream (B1, 106C), Auxiliary audio stream (B2), Auxiliary audio stream (B3), Metadata information capable of indicating that the audio element 152B from audio stream B2 is a general audio element 152B that also belongs to audio scene A.

[0239] · Audio scene C is encoded into three streams (C1, C2, and C3).

[0240] The audio streams 106 (106A, 106B, 106C...) are encoded at different bitrates (i.e. different representations) to allow for efficient bitrate adaptation, e.g. depending on the network connection (i.e. a higher bitrate coded version is delivered to users with a fast connection and a lower bitrate version is delivered to users with a slower network connection).

[0241] The audio streams 106 are stored on the media server 120 and for each audio stream, different encodings at different bitrates (i.e. different representations) are grouped into one adaptation set and appropriate data is generated to signal the availability of all adaptation sets. (Multiple representations of streams associated with the same audio signal may be present in the same adaptation set, but at different bitrates and / or qualities and / or resolutions.) Furthermore, in addition to the adaptation sets, the media server 120 can receive information about the location of the "boundary" of each audio scene and its relationship to each adaptation set (e.g., including a complete audio scene or only individual objects). In this way, each adaptation set can be associated with one or more of the available audio scenes 150. The boundary of one audio scene may for example be defined as the geometric coordinates of a sphere (e.g., center and radius).

[0242] o Each adaptation set may also contain descriptive information about where a sound scene or audio element 152 is active. For example, if one auxiliary stream (e.g., A2, 106B) contains one or more objects, the adaptation set may contain information such as where the objects can be heard (e.g., the coordinates and radius of the center of a sphere).

[0243] o Additionally or alternatively, each adaptation set (e.g., an adaptation set associated with scene B) may include descriptive information (e.g., metadata) that may indicate that audio elements (e.g., 152B) of one audio scene (e.g., B) are encoded into an audio stream (e.g., 106B) belonging to another audio scene (e.g., A) (also, or in addition).

[0244] The media server 120 can provide information to the system 102 (client), e.g., a DASH client, about the location of the "boundaries" associated with each adaptation set. For example, in the case of a DASH delivery environment, this may be embedded in the Media Presentation Description (MPD) XML syntax.

[0245] The system 102 (client) may receive information regarding the user's position and / or orientation and / or direction of movement (or information characterizing changes triggered by the user's actions).

[0246] The system 102 (client) can receive information about each adaptation set and based on this and / or the user's position and / or orientation and / or direction of movement (or information characterizing changes triggered by the user's actions, such as x, y, z coordinates or yaw, pitch, roll values), the system 102 (client) can select one or more adaptation sets that fully or partially describe the audio scene 150 to be played at the current location of the user 140.

[0247] · The system 102 (client) can request one or more adaptation sets.

[0248] o Furthermore, the system 102 (client) may select one or more adaptation sets that fully or partially describe the multiple audio scenes 150 and use the audio streams 106 corresponding to the multiple audio scenes 150 to create a new audio scene 150 that is played at the current location of the user 140.

[0249] o Based on metadata indicating that an audio element 152 is part of multiple audio scenes 150, the common audio element 152 can be requested only once to create a new audio scene instead of twice for each complete audio scene.

[0250] o Once the audio streams are available to the client system 102, in examples, one or more media decoders (104) may be used to decode the individual audio streams and / or additional mixers / renderers may be used to play the final audio scene based on information about the user's position and / or orientation and / or direction of movement (i.e., for example, disabling or not rendering some of the audio elements that cannot be heard at that particular location).

[0251] o alternatively or additionally, by using a metadata processor to manipulate audio metadata associated with all audio streams based on information about the user's position and / or orientation and / or direction of movement, Selecting / activating the required audio elements 152 (152A-152c) that compose the new audio scene; and / or · Allow merging all audio streams into a single audio stream.

[0252] · The media server 120 is able to deliver the necessary adaptation sets.

[0253] Alternatively, the system 102 (client) provides information about the positioning of the user 140 to the media server 120, which provides instructions about the required adaptation set.

[0254] Figure 1.5 shows another exemplary embodiment of such a system.

[0255] Encoding side A number of media encoders 154 can be used to create one or more audio streams 106 that embed audio elements 152 from one or more available audio scenes 150 associated with one sound scene portion of one viewpoint.

[0256] For each audio scene 150, one main stream can be created that contains only audio elements 152 that are part of the respective audio scene 150 and are not part of other audio scenes.

[0257] Additional auxiliary streams can be created for the same audio scene (e.g. some audio objects containing different languages ​​can be encoded into independent streams for efficient delivery).

[0258] Additional auxiliary streams can be created, including:

[0259] · An audio element 152 that is common to multiple audio scenes 150.

[0260] Metadata information indicating the association of this auxiliary stream with all other audio scenes 150 that share a common audio element 152. Or in other words, the metadata indicates the possibility that some audio streams can be used together with multiple audio scenes.

[0261] o Multiple media encoders that can be used to create one or more video streams for each available video scene associated with one video scene portion of one viewpoint. For simplicity, the video encoders are not shown in the figure.

[0262] o Media server 120 stores multiple audio and video adaptation sets that contain different encodings of the same audio and video streams at different bitrates (i.e., different representations). Additionally, media server 120 stores description information of every adaptation set, which may include:

[0263] Availability of all created adaptation sets.

[0264] Information describing the association of one adaptation set with one audio scene and / or viewpoint. In this way, each adaptation set can be associated with one of the available audio scenes.

[0265] Information describing the "boundaries" of each audio scene and / or viewpoint (which may, for example, contain complete audio scenes or only individual objects). The boundary of one audio scene may, for example, be defined as the geometric coordinates of a sphere (e.g. center and radius).

[0266] Information indicating the association of an adaptation set with multiple audio scenes that share at least one common audio element.

[0267] On the client side, a system (client system) that includes any of the following:

[0268] o A receiver capable of receiving: Information about the user's position and / or orientation and / or direction of movement (or information characterizing changes triggered by the user's actions) Information about the availability of all Adaptation Sets, as well as information describing the association of an Adaptation Set with an audio scene and / or viewpoint, and / or information describing the "boundaries" of each audio scene and / or viewpoint (which may, for example, include complete audio scenes or only individual objects). For example, such information may be provided as part of the Media Presentation Description (MPD) XML syntax in the case of a DASH delivery environment.

[0269] Information indicating the association of an adaptation set with multiple audio scenes that share at least one common audio element.

[0270] On the media consumption device side, used for content consumption (e.g., based on an HMD), the media consumption device is also responsible for collecting information about the user's position and / or orientation and / or direction of movement (or information characterizing changes triggered by the user's actions).

[0271] o The viewport processor 1232 can be configured as follows:

[0272] · Receive information about the current viewport, including the user's position and / or orientation and / or direction of movement, from the media consumption device (or information characterizing changes triggered by the user's actions).

[0273] Receive information about the ROI signaled in the metadata (video viewport signaled in the OMAF specification).

[0274] · Receive all information available at the receiving end.

[0275] Deciding which audio / video viewpoint to play at a particular moment based on all information received and / or derived from received and / or available metadata. For example, the viewport processor 1232 may decide:

[0276] One complete audio scene is recreated You need to create one new Audio Scene from all available Audio Scenes (for example, only some audio elements from multiple Audio Scenes are played, while other remaining audio elements from these Audio Scenes are not played).

[0277] The transition between two or more audio scenes needs to be reproduced o a selection part 1230 configured to select, based on information received from the viewport processor 1232, one or more adaptation sets from the available adaptation sets signalled in the information received by the receiving end, where the selected adaptation set fully or partially describes an audio scene to be played at the user's current location, which may be one or a partially complete audio scene defined on the encoding side, or a new audio scene has to be created from all available audio scenes.

[0278] Furthermore, an audio element 152 belonging to multiple audio scenes is selected based on information indicating an association between at least one adaptation set and multiple audio scenes containing the same audio element 152.

[0279] Further, based on instructions from the viewport processor 1232, if a transition between two or more audio scenes is about to occur, the selection portion 1230 can be configured to select one or more adaptation sets from the available adaptation sets signaled in the information received by the receiving end, where the selected adaptation set completely describes the audio scenes that need to be reproduced in the near future (e.g., if a user walks at a certain speed in the direction of the next audio scene, the next audio scene is predicted to be needed and selected prior to playback).

[0280] Furthermore, some adaptation sets corresponding to neighboring locations may be selected initially at a lower bitrate (i.e., a representation encoded at a lower bitrate is selected from the representations available in one adaptation set), and then, based on the change in location, the quality is improved by selecting a higher bitrate for these particular adaptation sets (i.e., a representation encoded at a higher bitrate is selected from the representations available in one adaptation set).

[0281] o The download and switching part, which can be configured as follows: · Requesting one or more of the available adaptation sets from the media server 120 based on the indication received from the selection portion 1230.

[0282] · Receive one or more adaptation sets from the available adaptation sets from the media server 120 (i.e. one representation out of all representations available within each adaptation set).

[0283] Extract metadata information from all received audio streams A metadata processor 1236 that can be configured as follows: Receiving information from download and switching information about the received audio streams, which may include audio metadata corresponding to each received audio stream.

[0284] by processing and manipulating audio metadata associated with each audio stream based on information received from the viewport processor 1232, which may include information regarding the user's position and / or orientation and / or direction of movement; · Select / activate the required audio elements 152 that compose the new audio scene, as indicated by the viewport processor 1232.

[0285] All audio streams can be merged into a single audio stream.

[0286] o The stream muxer / merger 1238 may be configured to merge all selected audio streams into one audio stream based on information received from the metadata processor 1236, which may include modified and processed audio metadata corresponding to all received audio streams.

[0287] o The media decoder is configured to receive and decode at least one audio stream for playback of a new audio scene, as indicated by the viewport processor 1232, based on information regarding the user's position and / or orientation and / or direction of movement.

[0288] FIG. 1.6 shows a system including a system (client system) which may embody, on the client side, for example, part of the system described in FIG. 5, which additionally or alternatively includes:

[0289] · Multiple media decoders can be configured to decode individual audio streams as indicated by the Metadata Processor 1236 (e.g. some audio elements are deactivated).

[0290] The mixer / renderer 1238 may be configured to play back the final audio scene based on information about the user's position and / or orientation and / or direction of movement (i.e., for example, disabling or not rendering some audio elements that cannot be heard at that particular location).

[0291] Update file format for file playback For file format use cases, multiple main and auxiliary streams can be encapsulated in a single ISOBMFF file as separate tracks. A single track in such a file represents a single audio element as described above. Since an MPD containing the information required for correct playout is not available, the information needs to be provided at the file format level, for example by providing / introducing specific file format boxes or specific file format boxes at the track and movie level. Depending on the use case, there are different pieces of information required to enable correct rendering of the encapsulated audio scene, but the following set of information is fundamental and must always be present:

[0292] Information about the contained audio scene, e.g. "location boundaries" All available audio elements, especially information about which audio elements are encapsulated in which tracks Location information for encapsulated audio elements A list of all audio elements that belong to one audio scene, an audio element may belong to multiple audio scenes.

[0293] With this information, all mentioned use cases should also work in a file-based environment, including those using additional metadata processors and shared encodings.

[0294] Further considerations regarding the above example: In an example (e.g., at least one of Figures 1.1-6), at least one scene can be associated with at least one audio element (audio source 152), each audio element associated with a position and / or area in the visual environment from which the audio element is heard, such that different audio streams are provided from server system 120 to client system 102 for different user positions and / or viewports and / or head orientations and / or movement data and / or interaction metadata and / or virtual position data within the scene.

[0295] In an example, the client system 102 may be configured to determine whether to play at least one audio element 152 and / or one adaptation set of an audio stream (e.g., A1, A2) in the presence of a current user's viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position in a scene, and the client system 102 is configured to request and / or receive the at least one audio element at the current user's virtual position.

[0296] In an example, a client system (e.g., 102) may be configured to predictively determine whether at least one audio element (152) and / or one adaptation set of an audio stream will be relevant and / or audible based on at least the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (110), the system configured to request and / or receive at least one audio element and / or audio stream and / or adaptation set at a particular user's virtual position before a predicted user movement and / or interaction in a scene, and the system configured to play the at least one audio element and / or audio stream upon receipt at a particular user's virtual position after the user movement and / or interaction in a scene. See, e.g., FIG. 8A and FIG. 8B above. In some examples, at least one of the operations of the system 102 or 120 may be performed based on the predictive data and / or statistical data and / or aggregated data.

[0297] In an example, a client system (e.g., 102) may be configured to request and / or receive at least one audio element (e.g., 152) at a lower bitrate and / or quality level at a virtual location of the user before the user's movement and / or interaction in the scene, and the system is configured to request and / or receive at least one audio element at a higher bitrate and / or quality level at a virtual location of the user after the user's movement and / or interaction in the scene. See, e.g., FIG. 7B.

[0298] In an example, at least one audio element may be associated with at least one scene, and at least one audio element may be associated with a location and / or region in the visual environment associated with the scene, and the system is configured to request different streams at different bitrates and / or quality levels of the audio elements based on a relevance and / or auditability level at each user's virtual location in the scene, the system is configured to request audio streams at higher bitrates and / or quality levels for audio elements that are more relevant and / or more audible at the current user's virtual location, and / or to request audio streams at lower bitrates and / or quality levels for audio elements that are less relevant and / or less audible at the current user's virtual location. Generally, see FIG. 7A. See also Figures 2a and 2b (the more relevant and / or more audible sources may be closer to the user), Figure 3 (the more relevant and / or more audible sources are the sources in scene 150a when the user is at position x1, and the more relevant and / or more audible sources are the sources in scene 150b when the user is at position x3), Figure 4 (at time t2, the more relevant and / or more audible sources may be from the first scene), and Figure 6 (the more audible sources may be the ones the user views directly).

[0299] In an example, at least one audio element (152) is associated with a scene, each audio element associated with a location and / or region within the visual environment associated with the scene, and the client system 102 is configured to periodically transmit the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (110) to the server system 120, such that locations closer to the at least one audio element (152) are provided with a higher bitrate and / or quality from the server, and locations further away from the at least one audio element (152) are provided with a lower bitrate and / or quality stream from the server. See, for example, Figures 2a and 2b.

[0300] In an example, multiple scenes (e.g., 150A, 150B) may be defined for multiple visual environments, such as adjacent and / or proximate environments, and a first stream associated with a first current scene (e.g., 150A) is provided, and when a user transitions (150AB) to a second further scene (e.g., 150B), both a stream associated with the first scene and a second stream associated with the second scene are provided. See, e.g., FIG. 3.

[0301] In an example, multiple scenes are defined for a first and a second visual environment, the first and second environments being adjacent and / or proximate environments, a first stream associated with the first scene is provided by the server for playback of the first scene when the user's virtual location is in the first environment associated with the first scene, a second stream associated with the second scene is provided by the server for playback of the second scene when the user's virtual location is in the second environment associated with the second scene, and both the first stream associated with the first scene and the second stream associated with the second scene are provided when the user's virtual location is in a transitional location between the first and second scenes. See, e.g., FIG. 3.

[0302] In the example, the first stream associated with the first scene is obtained at a higher bitrate and / or quality when the user is in the first environment associated with the first scene, while the second stream associated with the second scene environment associated with the second environment is obtained at a lower bitrate and / or quality when the user is at the start of the transition position from the first scene to the second scene. When the user is at the end of the transition position from the first scene to the second scene, the first stream associated with the first scene is obtained at a lower bitrate and / or quality, and the second stream associated with the second scene is obtained at a higher bitrate and / or quality. This is the case, for example, in Figure 3.

[0303] In the example, a plurality of scenes (e.g., 150A, 150B) are defined for a plurality of visual environments (e.g., adjacent environments), and the system 102 can request and / or obtain a stream associated with the current scene at a higher bitrate and / or quality and a stream associated with the second scene at a lower bitrate and / or quality. See, for example, Figure 4.

[0304] In the example, a plurality of N audio elements are defined. When the distance of the user to the position or area of these audio elements is greater than a predetermined threshold, the N audio elements are processed to obtain a smaller number M (M < N) of audio elements associated with a position or area closer to the position or area of the N audio elements, so that when the distance of the user to the position or area of the N audio elements is less than a predetermined threshold, at least one audio stream associated with the N audio elements is provided to the system, or when the distance of the user to the position or area of the N audio elements is greater than a predetermined threshold, at least one audio stream associated with the M audio elements is provided to the system. See, for example, Figure 1.7.

[0305] In an example, at least one visual environment scene is associated with at least one plurality of N audio elements (N >= 2), each audio element is associated with a position and / or region within the visual environment, at least one plurality of N audio elements may be provided in at least one representation at a high bitrate and / or quality level, at least one plurality of N audio elements may be provided in at least one representation at a low bitrate and / or quality level, at least one representation is obtained by processing the N audio elements to obtain a smaller number M (M < N) of audio elements associated with a position or region close to the position or region of the N audio elements, the system is configured to request a representation at a higher bitrate and / or quality level for an audio element if the audio element is more relevant and / or more audible at the current virtual position of the user in the scene, and the system is configured to request a representation at a lower bitrate and / or quality level for an audio element if the audio element is less relevant and / or less audible at the current virtual position of the user in the scene. For example, see Figure 1.7.

[0306] In an example, different streams are obtained for different audio elements when the user's distance and / or relevance and / or audible level and / or angular orientation is below a predetermined threshold. For example, see Figure 1.7.

[0307] In an example, since different audio elements are provided in different viewports, if a first audio element is within the current viewport, the first audio element is obtained at a higher bitrate than a second audio element that is not within the viewport. For example, see Figure 6.

[0308] In an example, at least two visual environment scenes are defined, at least one first and second audio element is associated with a first scene associated with the first visual environment, and at least one third audio element is associated with a second scene associated with the second visual environment, the system 102 is configured to obtain metadata describing that the at least one second audio element is further associated with the second visual environment scene, the system is configured to request and / or receive at least the first and second audio elements when the user's virtual location is in the first visual environment, the system is configured to request and / or receive at least the second and third audio elements when the user's virtual location is in the second visual environment scene, and the system is configured to request and / or receive at least the first and second and third audio elements when the user's virtual location is transitioning between the first visual environment scene and the second visual environment scene. See, for example, FIG. 1.4. This also applies to FIG. 3.

[0309] In an example, the at least one first audio element may be provided in at least one audio stream and / or adaptation set, the at least one second audio element is provided in at least one second audio stream and / or adaptation set, the at least one third audio element is provided in at least one third audio stream and / or adaptation set, at least a first visual environment scene is described by the metadata as a complete scene requiring at least the first and second audio streams and / or adaptation sets, the second visual environment scene is described by the metadata as an incomplete scene requiring at least a third audio stream and / or adaptation set, as well as at least a second audio stream and / or adaptation set associated with the at least first visual environment scene, and the system includes a metadata processor configured to manipulate the metadata to enable merging of the second audio stream belonging to the first visual environment and the third audio stream associated with the second visual environment into a new single stream when the user's virtual location is in the second visual environment. See, for example, Figures 1.2-1.3, 1.5, and 1.6.

[0310] In an example, the system 102 may include a metadata processor (e.g., 1236) configured to manipulate metadata in the at least one audio stream prior to the at least one audio decoder based on a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data.

[0311] In an example, the metadata processor (e.g., 1236) may be configured to enable and / or disable at least one audio element in the at least one audio stream before the at least one audio decoder based on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, and if the system determines that the audio element is no longer played as a result of the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, the metadata processor may be configured to disable the at least one audio element in the at least one audio stream before the at least one audio decoder, and if the system determines that the audio element is played as a result of the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, the metadata processor may be configured to enable the at least one audio element in the at least one audio stream before the at least one audio decoder.

[0312] Server Side Reference is also made herein to a server (120) for delivering audio and video streams to a client for a virtual reality VR, augmented reality AR, mixed reality MR or 360 degree video environment, the video and audio streams being played on a media consumption device, the server (120) comprising an encoder for encoding and / or a storage device for storing a video stream describing a visual environment, the visual environment being associated with an audio scene, the server further comprising an encoder for encoding and / or a storage device for storing a plurality of streams and / or audio elements and / or adaptation sets to be delivered to the client, the streams and / or audio elements and / or adaptation sets being associated with at least one audio scene, the server further comprising: Selecting and delivering video streams based on client requests, The video stream is associated with the environment, and the audio stream and / or audio element and / or adaptation set are selected based on a request from a client, the request being associated with at least a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, and an audio scene associated with the environment; The audio stream is configured to be delivered to a client.

[0313] Further embodiments and variations Depending on the particular implementation, the embodiments may be implemented in hardware. The embodiments may be implemented using digital storage media, such as, for example, floppy disks, digital versatile disks (DVDs), Blu-ray disks, compact disks (CDs), read-only memories (ROMs), programmable read-only memories (PROMs), erasable and programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs) or flash memories, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system to perform the respective methods. Thus, the digital storage media may be computer readable.

[0314] In general, the embodiments may be implemented as a computer program product including program instructions that operate to perform one of the methods when the computer program product is executed on a computer. The program instructions may be stored, for example, on a machine-readable medium.

[0315] Other embodiments include the computer program for performing one of the methods described herein, stored on a machine-readable carrier. In other words, an example of a method is therefore the computer program having program instructions for performing one of the methods described herein, when the computer program runs on a computer.

[0316] A further example of a method is therefore a data carrier medium (or digital storage medium, or computer readable medium) comprising a computer program for performing one of the methods described herein and having recorded thereon. The data carrier medium, digital storage medium, or recorded medium is not an intangible, transitory signal, but is tangible and / or non-transitory.

[0317] Further examples include a processing unit, such as a computer, or a programmable logic device, that performs one of the methods described herein.

[0318] A further example comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0319] Further examples include an apparatus or system that transfers (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0320] In some examples, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some examples, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods may be performed by any suitable hardware apparatus.

[0321] Further examples include: [1] A system (102) for a virtual reality (VR), augmented reality (AR), mixed reality (MR), or 360-degree video environment configured to receive video and audio streams to be played on a media consumption device, comprising: The system (102) comprises: at least one media video decoder configured to decode a video signal from the video stream to represent a VR, AR, MR, or 360 degree video environment scene to a user; at least one audio decoder (104) configured to decode an audio signal (108) from at least one audio stream (106); The system (102) is configured to request (112) at least one audio stream (106) and / or one audio element of an audio stream and / or one adaptation set from a server (120) based on at least a current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (110) of the user. Further examples include: [2] The system described in [1], configured to provide the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (110) to the server (120) in order to obtain the at least one audio stream (106) and / or one audio element of the audio stream and / or one adaptation set from the server (120). Further examples include: [3] The system described in [1] or [2], wherein at least one scene is associated with at least one audio element (152), each audio element being associated with a position and / or area in the visual environment where the audio element is audible, and different audio streams are provided for different user positions and / or viewports and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data within the scene. Further examples include: [4] configured to determine whether to play at least one audio element and / or one adaptation set of the audio stream with respect to a current user viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position in the scene; The system is a system described in any one of [1] to [3], configured to request and / or receive the at least one audio element at the current user's virtual location. Further examples include: [5] the system is configured to predictively determine whether at least one audio element (152) of an audio stream and / or one adaptation set will be relevant and / or audible based on at least a current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (110) of the user; the system is configured to request and / or receive the at least one audio element and / or audio stream and / or adaptation set at a specific user virtual position prior to the predicted user movement and / or interaction in the scene; The system of any one of claims 1 to 4, wherein upon receipt, the system is configured to play the at least one audio element and / or audio stream at a virtual position of the particular user after the user's movement and / or interaction in the scene. Further examples include: [6] configured to request and / or receive the at least one audio element (152) at a lower bitrate and / or quality level at a virtual position of the user prior to the user's movement and / or interaction in the scene; The system of any one of claims [1] to [5], wherein the system is configured to request and / or receive the at least one audio element at a higher bitrate and / or quality level at a virtual position of the user after the user's movement and / or interaction in the scene. Further examples include: [7] At least one audio element (152) is associated with at least one scene, each audio element being associated with a location and / or area within the visual environment associated with the scene; The system described in any one of [1] to [6] is configured to request and / or receive streams at a higher bit rate and / or quality for audio elements closer to the user than for audio elements farther from the user. Further examples include: [8] At least one audio element (152) is associated with at least one scene, the at least one audio element being associated with a location and / or area within the visual environment associated with the scene; the system is configured to request different streams at different bit rates and / or quality levels of audio elements based on a relevance and / or auditability level at each user's virtual location in the scene; the system is configured to request audio streams at higher bit rates and / or quality levels for audio elements that are more relevant and / or more audible at the current user's virtual location; and / or The system of any one of claims 1 to 7, configured to request audio streams at lower bit rates and / or quality levels for audio elements that are less relevant and / or less audible at the current user's virtual location. Further examples include: [9] At least one audio element (152) is associated with a scene, each audio element being associated with a location and / or area within the visual environment associated with the scene; The system is configured to periodically transmit the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (110) to the server, whereby: In a first location, a higher bit rate and / or quality stream is provided by the server; In a second location, a lower bit rate and / or quality stream is provided by the server; The system according to any one of claims [1] to [8], wherein the first position is closer to the at least one audio element (152) than the second position. Further examples include:

[10] A plurality of scenes (150A, 150B) are defined for a plurality of visual environments, such as adjacent and / or proximate environments; The system of any one of claims [1] to [9], wherein a first stream associated with a first current scene is provided, and when a user transitions to a second further scene, both the stream associated with the first scene and the second stream associated with the second scene are provided. Further examples include:

[11] A plurality of scenes (150A, 150B) are defined for a first and a second visual environment, the first and second environments being adjacent and / or proximate environments; a first stream associated with the first scene is provided from the server for playback of the first scene when the user's location or virtual location is in a first environment associated with the first scene; a second stream associated with the second scene is provided by the server for playback of the second scene when the user's location or virtual location is in a second environment associated with the second scene; The system described in any one of [1] to

[10] , wherein when the user's position or virtual position is in a transition position between the first scene and the second scene, both a first stream associated with the first scene and a second stream associated with the second scene are provided. Further examples include:

[12] A plurality of scenes (150A, 150B) are defined for first and second visual environments, the first and second visual environments being adjacent and / or proximate environments; the system is configured to request and / or receive a first stream associated with a first scene associated with the first environment for playback of the first scene (150A) when the user's virtual location is in the first environment; the system is configured to request and / or receive a second stream associated with the second scene associated with the second environment for playback of the second scene (150B) when the user's virtual location is in the second environment; The system is the system described in any one of [1] to

[11] , configured to request and / or receive both a first stream associated with the first scene and a second stream associated with the second scene when the user's virtual position is at a transition position (150AB) between the first environment and the second environment. Further examples include:

[13] The first stream associated with the first scene is acquired at a higher bit rate and / or quality when the user is in the first environment associated with the first scene; Meanwhile, the second stream associated with the second scene associated with the second environment is acquired at a lower bitrate and / or quality when the user is at the beginning of a transition position from the first scene to the second scene, when the user is at the end of a transition position from the first scene to the second scene, the first stream associated with the first scene is acquired at a lower bitrate and / or quality and the second stream associated with the second scene is acquired at a higher bitrate and / or quality; The system according to any one of claims

[10] to

[12] , wherein the lower bit rate and / or quality is lower than the higher bit rate and / or quality. Further examples include:

[14] A plurality of scenes (150A, 150B) are defined for a plurality of environments, such as adjacent and / or neighboring environments; The system is configured to obtain the stream associated with a first current scene associated with a first current environment; The system of any one of claims [1] to

[13] , wherein if the distance of the user's position or virtual position from the boundary of the scene is less than a predetermined threshold, the system further acquires an audio stream associated with a second adjacent and / or nearby environment associated with the second scene. Further examples include:

[15] A plurality of scenes (150A, 150B) are defined for a plurality of visual environments; the system requests and / or obtains the stream associated with the current scene at a higher bitrate and / or quality and the stream associated with the second scene at a lower bitrate and / or quality; The system according to any one of claims [1] to

[14] , wherein the lower bit rate and / or quality is lower than the higher bit rate and / or quality. Further examples include: 〔16〕A plurality of N audio elements are defined, and when the distance of the user to the position or region of these audio elements is greater than a predetermined threshold value, the N audio elements are processed to obtain a smaller number M (M < N) of audio elements associated with a position or region close to the position or region of the N audio elements, whereby, when the distance of the user to the position or region of the N audio elements is smaller than a predetermined threshold value, at least one audio stream associated with the N audio elements is provided to the system, or when the distance of the user to the position or region of the N audio elements is greater than a predetermined threshold value, at least one audio stream associated with the M audio elements is provided to the system, which is the system according to any one of 〔1〕to 〔15〕above. Also, a further example is, 〔17〕At least one visual environment scene is associated with at least one plurality of N audio elements (N >= 2), and each audio element is associated with a position and / or region within the visual environment, the at least one plurality of N audio elements are provided in at least one representation at a high bit rate and / or quality level, the at least one plurality of N audio elements are provided in at least one representation at a low bit rate and / or quality level, and the at least one representation processes the N audio elements to obtain a smaller number M (M <n)個のオーディオ要素を取得することによって取得され、the system is configured to request the representation at a higher bitrate and / or quality level for an audio element if the audio element is more relevant and / or more audible at the user's current virtual position in the scene; The system of any one of claims 1 to 16, wherein the system is configured to request the representation at a lower bitrate and / or quality level for an audio element if the audio element is less relevant and / or less audible at the user's current virtual position in the scene. Further examples include:

[18] The system described in

[16] and

[17] , wherein different streams are obtained for the different audio elements when the user's distance and / or relevance and / or audibility level and / or angular orientation are lower than a predetermined threshold. Further examples include:

[19] The system of any one of [1] to

[18] , wherein the system is configured to request and / or obtain the stream based on the user's orientation in the scene and / or the direction of the user's movement and / or the user's interaction. Further examples include:

[20] A system described in any one of [1] to

[19] , wherein the viewport is associated with the position and / or virtual position and / or movement data and / or head. Further examples include:

[21] A system as described in any one of [1] to

[20] , wherein different audio elements are provided in different viewports, and the system is configured, when one first audio element (S1) is within a viewport (160-1), to request and / or receive a first audio element of a higher bitrate than a second audio element (S2) that is not within the viewport. Further examples include:

[22] A system is configured to request and / or receive a first audio stream and a second audio stream, the first audio element of the first audio stream being more relevant and / or more audible than the second audio element of the second audio stream; The system described in any one of [1] to

[21] , wherein the first audio stream is requested and / or received at a bitrate and / or quality higher than the bitrate and / or quality of the second audio stream. Further examples include:

[23] At least two visual environment scenes are defined, and at least one first and second audio element is associated with a first scene associated with the first visual environment, and at least one third audio element is associated with a second scene associated with the second visual environment; the system is configured to obtain metadata describing at least one second audio element further associated with a second visual environment scene; the system is configured to request and / or receive the at least first and second audio elements when the user's virtual location is in the first visual environment; the system is configured to request and / or receive the at least second and third audio elements when the user's virtual location is in the second visual environment scene; The system is the system described in any one of [1] to

[22] , wherein the system is configured to request and / or receive the at least first, second and third audio elements when the user's virtual position is transitioning between the first visual environment scene and the second visual environment scene. Further examples include:

[24] The at least one first audio element is provided in at least one audio stream and / or adaptation set, the at least one second audio element is provided in at least one second audio stream and / or adaptation set, the at least one third audio element is provided in at least one third audio stream and / or adaptation set, the at least one first visual environment scene is described by metadata as a complete scene requiring the at least one first and second audio streams and / or adaptation set, and the second visual environment scene is described by metadata as an incomplete scene requiring the at least one third audio stream and / or adaptation set, and the at least one second audio stream and / or adaptation set associated with the at least one first visual environment scene; The system of claim 23, further comprising a metadata processor configured to manipulate the metadata to enable merging of the second audio stream belonging to the first visual environment and the third audio stream associated with the second visual environment into a new single stream when the user's virtual location is in the second visual environment. Further examples include:

[25] The system of any one of [1] to

[24] , further comprising a metadata processor configured to manipulate the metadata in at least one audio stream prior to the at least one audio decoder based on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data. Further examples include:

[26] The metadata processor is configured to enable and / or disable at least one audio element in at least one audio stream prior to the at least one audio decoder based on a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data; wherein the metadata processor is configured to disable at least one audio element in the at least one audio stream prior to the at least one audio decoder when the system determines that the audio element is not to be played anymore as a result of a current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data; The system of

[25] , wherein the metadata processor is configured to enable at least one audio element in at least one audio stream before the at least one audio decoder when the system determines that the audio element is to be played as a result of a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data. Further examples include:

[27] A system described in any one of [1] to

[26] , configured to disable decoding of selected audio elements based on the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position. Further examples include:

[28] A system described in any one of [1] to

[27] , configured to merge at least one first audio stream associated with the current audio scene with at least one stream associated with an adjacent, nearby, and / or future audio scene. Further examples include:

[29] A system described in any one of [1] to

[28] , configured to obtain and / or collect statistical or aggregated data regarding the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data, and send a request to the server associated with the statistical or aggregated data. Further examples include:

[30] The system described in any one of [1] to

[29] , configured to deactivate decoding and / or playback of at least one stream based on metadata associated with the at least one stream and based on the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data. Further examples include:

[31] Manipulating metadata associated with the selected group of audio streams based on at least the user's current or estimated viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data; Selecting and / or enabling and / or activating the audio elements that make up said audio scene to be played, and / or The system according to any one of claims 1 to 30, further configured to enable merging of all selected audio streams into a single audio stream. Further examples include:

[32] A system as described in any one of [1] to

[31] , configured to control the request for the at least one stream to the server based on a distance of the user's position from the boundaries of adjacent and / or nearby environments associated with different scenes, or other metric associated with the user's position in the current environment or a prediction of the user's position in the future environment. Further examples include:

[33] A system described in any one of [1] to

[32] , wherein for each audio element or audio object, information is provided from the server system (120), the information including descriptive information about the sound scene or where the audio element is active. Further examples include:

[34] The system of any one of [1] to

[33] , configured to select between playing one scene and compositing, mixing, multiplexing, superimposing or combining at least two scenes based on the current or future or viewport and / or head orientation and / or movement data and / or metadata and / or virtual position and / or user selection, the two scenes being associated with different adjacent and / or nearby environments. Further examples include:

[35] A method for generating or using at least the adaptation set, Several Adaptation Sets are associated with one Audio Scene, and / or Additional information is provided that associates each adaptation set with one viewpoint or one audio scene; and / or information about said boundaries of an audio scene, and / or Information about the relationship between an adaptation set and an audio scene (e.g., an audio scene is encoded into three streams encapsulated in three adaptation sets), and / or information regarding connections between the boundaries of the audio scene and the plurality of adaptation sets; The system according to any one of claims 1 to 34, further comprising: Further examples include:

[36] receiving a stream of scenes associated with an adjacent or proximate environment; upon detection of the transition of the boundary between two environments, starting the decoding and / or playing of the stream of the adjacent or proximate environment. The system according to any one of claims [1] to

[35] , configured as follows: Further examples include:

[37] A system comprising: the system described in any one of [1] to

[36] configured to operate as a client; and a server configured to deliver video and / or audio streams to be played on a media consumption device. Further examples include:

[38] The system comprises: Requesting and / or receiving at least one first adaptation set including at least one audio stream associated with at least one first audio scene; requesting and / or receiving at least one second adaptation set including at least one second audio stream associated with at least two audio scenes including the at least one first audio scene; enabling merging of the at least one first audio stream and the at least one second audio stream into a new audio stream to be decoded based on available metadata regarding a user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data and / or information describing an association of the at least one first adaptation set to the at least one first audio scene and / or an association of the at least one second adaptation set to the at least one first audio scene, The system according to any one of claims 1 to 37, further configured as follows: Further examples include:

[39] receiving information about a user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data, and / or information characterizing changes triggered by actions of the user; receiving information regarding the availability of an adaptation set and information describing an association of the at least one adaptation set to at least one scene and / or viewpoint and / or viewport and / or position and / or virtual position and / or motion data and / or orientation; The system according to any one of claims [1] to

[38] , configured as follows: Further examples include:

[40] Determine whether to play at least one audio element from at least one audio scene embedded in at least one stream and at least one additional audio element from at least one additional audio scene embedded in at least one additional stream; - in case of a positive determination, performing an operation of merging or synthesizing or multiplexing or superimposing or combining said at least one additional stream of said additional audio scene with said at least one stream of said at least one audio scene. The system according to any one of claims [1] to

[39] , configured as follows: Further examples include:

[41] Manipulating audio metadata associated with the selected audio stream based on at least the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data; selecting and / or enabling and / or activating said audio elements constituting said audio scene determined to be played; Allows merging all selected audio streams into a single audio stream, The system according to any one of claims [1] to

[40] , configured as follows: Further examples include:

[42] A server (120) for delivering audio and video streams for a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), or a 360-degree video environment to a client, the video and audio streams being played on a media consumption device; The server (120) includes an encoder for encoding and / or a storage device for storing a video stream describing a visual environment, the visual environment being associated with an audio scene; the server further comprises an encoder for encoding and / or a storage device for storing a plurality of streams and / or audio elements and / or adaptation sets to be delivered to the client, the streams and / or audio elements and / or adaptation sets being associated with at least one audio scene; The server, Selecting and delivering a video stream based on a request from the client, the video stream being associated with an environment; selecting an audio stream and / or an audio element and / or an adaptation set based on a request from the client, the request being associated with at least a current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data of the user and an audio scene associated with the environment; Delivering the audio stream to the client; The server (120) is configured as follows. Further examples include:

[43] The streams are encapsulated into adaptation sets, each adaptation set including multiple streams associated with different representations at different bit rates and / or qualities of the same audio content; The server according to

[42] , wherein the selected adaptation set is selected based on the request from the client. Further examples include:

[44] A system including the system described in any one of [1] to

[41] acting as a client and as the server. Further examples include:

[45] The system described in

[44] , including the server described in

[42] or

[43] . Further examples include:

[46] A method for a virtual reality (VR), augmented reality (AR), mixed reality (MR), or 360-degree video environment configured to receive a video stream and an audio stream to be played on a media consumption device, comprising: decoding a video signal from the video stream for presentation to a user of a VR, AR, MR, or 360 degree video environment scene; decoding an audio signal from the audio stream; - requesting and / or obtaining from a server at least one audio stream based on a current viewport and / or position data and / or head orientation and / or movement data and / or metadata and / or virtual position data and / or metadata of said user; The method includes: Further examples include:

[47] A computer program comprising instructions that, when executed by a processor, cause the processor to perform the method described in

[46] . The above examples are illustrative of the principles described above. It is understood that modifications and variations of the arrangements and details described herein are obvious. It is therefore intended to be limited by the scope of the appended claims and not by the specific details presented as descriptions and illustrations of the embodiments herein.

Claims

1. 1. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; A plurality of audio scenes (150A, 150B) are defined, The system (102) requests or receives the first audio stream or the first adaptation set associated with a first current audio scene at a higher bitrate and the second audio stream or the second adaptation set associated with a second audio scene at a lower bitrate.

2. 2. The system of claim 1, wherein one of the first and second audio streams or the first and second adaptation sets is requested or received based on a current audio scene regardless of user movement data, while the other one of the first and second audio streams or the first and second adaptation sets is requested or received based on user movement data.

3. The system of claim 1 , configured to determine the audibility of the audio elements in a current audio scene, the first audio element being an audio element having a higher audibility than the second audio element.

4. The system of claim 1 , configured to determine the relevance of the audio elements in a current audio scene, the first audio element being an audio element having a higher relevance than the second audio element.

5. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; A system in which the second audio stream or the second adaptation set is initially requested or received at a lower bit rate, and after the reception of user movement data reflecting higher relevance or audibility of the element in the audio scene, the system requests transmission of the second audio stream or the second adaptation set at a higher bit rate, and thereafter receives the first audio stream or the first adaptation set at a higher bit rate.

6. The system of claim 1 , configured to output a request based on current movement data of the user to obtain at least one audio stream (106) and / or one audio element of an audio stream and / or one adaptation set based on the movement data of the user.

7. 2. The system of claim 1, wherein at least one scene is associated with at least one audio element (152), each audio element being associated with a location and / or area within the virtual video environment, and wherein different audio streams are provided for different user movement data.

8. configured to determine whether to play at least one audio element and / or one adaptation set of the audio stream based on current user movement data or virtual position; The system of claim 1 , wherein the system is configured to request and / or receive the at least one audio element with current user virtual position and / or movement data.

9. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; the system is configured to predictively determine, based at least on current movement data of the user, whether at least one audio element (152) of an audio stream and / or an adaptation set will be relevant and / or audible; the system is configured to request and / or receive the at least one audio element and / or audio stream and / or adaptation set upon a specific user movement, The system is configured, upon receipt, to play the at least one audio element and / or audio stream following a movement and / or interaction of the user in an audio scene.

10. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; configured to request and / or receive the at least one audio element (152) at a lower bit rate in the second audio stream or in the second adaptation set, the interaction resulting from user movement data in the same audio scene or when a new audio scene is started; The system is configured to request and / or receive the at least one audio element at a higher bitrate in the first stream or the first adaptation set at the user's position in the audio scene and / or at the user's virtual position after interaction.

11. The system is configured to periodically output requests based on current movement data of the user, whereby: at least one first position or movement data is provided with the first audio stream at a higher bit rate; The system of claim 1 , wherein at least one of second position or movement data provides the second audio stream at a lower bit rate.

12. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; A plurality of audio scenes (150A, 150B) are defined, A system, wherein a first stream is associated with a first current audio scene, and when transitioning to a second further audio scene, both the first audio stream associated with the first current audio scene and the second audio stream associated with the second further audio scene are provided.

13. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; A plurality of audio scenes (150A, 150B) are defined including a first audio scene and a second audio scene; a first audio stream or a first adaptation set associated with a first audio scene is provided for playback of the first audio scene when the user's movement data requires playback of the first audio scene; a second audio stream or a second adaptation set associated with a second audio scene is provided for playback of the second audio scene when the user movement data requires playback of the second audio scene; When the user's movement data requires a transition between the first audio scene and the second audio scene, both a first audio stream or a first adaptation set associated with the first audio scene and a second audio stream or a second adaptation set associated with the second audio scene are provided.

14. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; A plurality of audio scenes (150A, 150B) are defined, the system is configured to request and / or receive the first audio stream or the first adaptation set associated with a first audio scene (150A) for playback of the first audio scene when the motion data requires playback of the first audio scene; the system is configured to request and / or receive a second audio stream associated with a second audio scene for playback of the second audio scene when the motion data requires playback of the second audio scene; The system is configured to request and / or receive both the first stream or the first adaptation set associated with the first audio scene and the second stream or the second adaptation set associated with the second audio scene when the user movement data requires a transition from the first audio scene to the second audio scene.

15. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; At the start of a transition from a first audio scene to a second audio scene, the system: At the higher bit rate, the first audio stream or the first adaptation set is associated with the first audio scene; and receiving or requesting, at the lower bit rate, that the second audio stream or the second adaptation set be associated with the second audio scene; At the end of the transition from the first audio scene to the second audio scene, the system: At the higher bit rate, the first audio stream or the first adaptation set is associated with a second audio scene; and At the lower bit rate, a second audio stream or the second adaptation set is associated with the first audio scene.

16. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; A plurality of N audio elements is defined, and if the user movement data requires the reproduction of the plurality of N audio elements, the plurality of N audio elements is processed to obtain a smaller number M of audio elements (M<N), whereby providing the system with the first audio stream or the first adaptation set associated with the plurality of N audio elements if the user movement data indicates that the N audio elements to be played are more relevant; or If the user movement data means that the plurality of N audio elements being played are more relevant, the system is provided with the second audio stream or the second adaptation set associated with the M audio elements.

17. At least one audio scene is associated with at least one plurality of N audio elements; At least one of said plurality of N audio elements is provided in at least one representation at a high bit rate and / or quality level, At least one plurality of N audio elements is provided in at least one representation at a low bit rate, said at least one representation being obtained by processing said N audio elements to obtain a smaller number M (M<N) of audio elements associated with positions or regions close to positions or regions of said N audio elements; the system is configured to request or receive a first stream or the first adaptation set at a higher bitrate for an audio element if the audio element is more relevant and / or more audible in the current audio scene, 2. The system of claim 1, wherein the system is configured to request or receive the first stream or the first adaptation set at a lower bitrate for an audio element if the audio element is less relevant and / or less audible.

18. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; At least two audio scenes are defined, at least one first and one second audio element is associated with the first audio scene, and at least one third audio element is associated with the second audio scene; the system is configured to obtain interaction metadata describing at least one second audio element further associated with the second audio scene; the system is configured to request and / or receive the at least first and second audio elements when the user movement data requires playback of the first audio scene; the system is configured to request and / or receive the at least second and third audio elements when the user movement data requires playback of the second audio scene; The system is configured to request and / or receive the at least first, second and third audio elements when the user movement data requires a transition between the first audio scene and a second audio scene.

19. the at least one first audio element is provided in at least one audio stream and / or adaptation set, the at least one second audio element is provided in at least one second audio stream and / or adaptation set, the at least one third audio element is provided in at least one third audio stream and / or adaptation set, and the second audio scene is described by interaction metadata as an audio scene using the at least one third audio stream and / or adaptation set and the at least second audio stream and / or adaptation set associated with the at least first audio scene; 20. The system of claim 18, further comprising a metadata processor configured to manipulate the interaction metadata to merge the second audio stream and the third audio stream into a new single stream when the user movement data requires playback of the second audio stream.

20. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; The system includes a metadata processor configured to manipulate metadata in at least one audio stream prior to the at least one audio decoder based on the user movement data.

21. the metadata processor is configured to enable and / or disable at least one audio element in at least one audio stream prior to the at least one audio decoder based on current movement data of a user; when the system determines that the audio element is no longer to be played as a result of the current motion data, the metadata processor is configured to disable at least one audio element in at least one audio stream prior to the at least one audio decoder; 21. The system of claim 20, wherein the metadata processor is configured to enable at least one audio element in at least one audio stream prior to the at least one audio decoder if the system determines that the audio element is to be played as a result of the current motion data.

22. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; The system is configured to disable decoding of selected audio elements based on current movement data of the user.

23. The system of claim 1 , configured to merge the first audio stream or the first adaptation set with the second audio stream or the second adaptation set.

24. 24. The system of claim 23, configured to merge the first audio stream or the first adaptation set with the second audio stream or the second adaptation set upstream of the at least one audio decoder to obtain one audio stream that is decoded by the at least one audio decoder.

25. The at least one audio decoder Decoding the first audio stream or the first adaptation set to obtain a first decoded audio signal; and decoding the second audio stream or the second adaptation set to obtain a second decoded audio signal; 25. The system of claim 24, further comprising a mixer that mixes the first decoded audio signal and the second decoded audio signal.

26. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; A system configured to obtain and / or collect statistical or aggregate data regarding the user movement data and to send requests associated with the statistical or aggregate data.

27. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; manipulating metadata associated with the selected group of audio streams based on at least the current movement data of the user; Selecting or activating the audio elements that make up said audio scene to be played, or The system is further configured to merge all the selected audio streams into a single audio stream.

28. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; A system in which, for each audio element or audio object, information is received, said information including descriptive information of which of the audio elements are active.

29. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; The system comprises: Requesting and / or receiving a first audio stream and / or a first adaptation set, the first audio stream and / or the first adaptation set including at least one audio stream associated with at least one first audio scene; requesting and / or receiving a second audio stream and / or a second adaptation set, the second audio stream including at least one second audio stream associated with at least a second audio scene; merging the at least one first audio stream and the at least one second audio stream into a new audio stream to be decoded based on available metadata related to user movement data and / or on information describing an association of the first audio stream and / or the first adaptation set to the at least one first audio scene and / or an association of the at least one second audio stream and / or the second adaptation set to the at least one first audio scene. The system further comprises:

30. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; receiving information regarding user movement data and / or information characterizing changes triggered by the user's actions; receiving information regarding availability of adaptation sets and information describing an association of at least one adaptation set to at least one scene; The system is configured as follows.

31. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; determining whether to play at least one audio element from at least one audio scene embedded in at least one of the first and second audio streams and at least one additional audio element from at least one additional audio scene embedded in at least one additional stream; - in case of a positive determination, performing an operation of merging or synthesizing or multiplexing or superimposing or combining said at least one additional stream of said additional audio scene with said at least one stream of said at least one audio scene. The system is configured as follows.

32. A system for receiving an audio stream to be played, comprising: The system comprises: at least one audio decoder for decoding an audio signal from the audio stream to represent an audio scene; The system comprises: a first audio stream or a first adaptation set; and a second audio stream or a second adaptation set; a first audio element in the first adaptation set is more relevant and / or more audible than a second audio element in the second adaptation set, and the first audio stream or the first adaptation set is requested or received at a higher bitrate than the second audio stream or the second adaptation set; Manipulating audio metadata associated with the selected audio stream based on at least the current movement data of the user; selecting and / or validating and / or activating the audio elements that compose the audio scene determined to be played, Allows merging all selected audio streams into a single audio stream, The system is configured as follows.

33. 2. The system of claim 1, wherein the first audio stream and the second audio stream are encapsulated in audio segments according to MPEG-2 or MPEG-4.

34. 1. A method for receiving a first audio stream and a second audio stream for playback on a media consumption device, comprising: decoding an audio signal from the audio stream for a representation of a plurality of audio scenes; - requesting and / or obtaining at least one audio stream based on the user's current movement data and / or metadata and / or virtual position data and / or metadata; Including, a first audio element in the first audio stream is more relevant or audible than a second audio element in the second audio stream, and the first audio stream is requested or received at a higher bit rate than a bit rate of the second audio stream; A plurality of audio scenes (150A, 150B) are defined, The method includes requesting or receiving the first audio stream associated with a first current audio scene at a higher bitrate and the second audio stream associated with a second audio scene at a lower bitrate.

35. A computer program comprising instructions which, when executed by a processor, cause the processor to carry out the method of claim 34.

Images